Ordered testing with XUnit, NUnit and MSTest part 5: NUnit implementation revised
We have previously implemented some basic ordered testing with NUnit. However, it had some severe limitations. The biggest limitation was that test fixture ordering across namespaces didn’t work. This made the implementation pretty useless. In this blog post we’re going to find a solution and implement it. The code of this blog post can be found on GitHub. Disclaimer: This code will have rough edges, and may not work for you, kill you cat or blow up in your face.
This post is part of a series of posts on ordered testing:
- Ordered testing with XUnit, NUnit and MSTest part 1: The setup
- Ordered testing with XUnit, NUnit and MSTest part 2: MSTest
- Ordered testing with XUnit, NUnit and MSTest part 3: NUnit
- Ordered testing with XUnit, NUnit and MSTest part 4: XUnit
- Ordered testing with XUnit, NUnit and MSTest part 5: NUnit implementation revised
- Ordered testing with XUnit, NUnit and MSTest part 6: NUnit implementation revised part 2
Test discovery in NUnit is done by an
ITestAssemblyBuilder implementation. The implementation is invoked by the
FrameworkController when the assembly is loaded.
The default and only implementation of the
ITestAssemblyBuilder interface is
DefaultTestAssemblyBuilder. This implementation discovers the tests and creates an hierarchy of test fixtures based on namespace.
DefaultTestAssemblyBuilder is hardcoded in NUnit.
FrameworkController does have a constructor accepting the fully qualified type name of a
ITestAssemblyBuilder, but this constructors isn’t used anywhere and cannot be used for our purposes. We have only one choice left, besides giving up: Fork NUnit.
Our goal is not to implement ordered testing right within NUnit. Let’s keep the differences between our fork and the main NUnit code base as small as possible. We want to be able to specify a custom
ITestAssemblyBuilder for an assembly. We’re going to make this configurable using an attribute:
Now we implement a
ITestAssemblyBuilder which simply instantiates and calls the correct
And in the
FrameworkController replace the hardcoded call to
That’s all the code we need to modify in the NUnit fork. This code now allows NUnit to use a different
ITestAssemblyBuilder implementation if supplied. The source code of the fork can be found on GitHub.
Designing the test ordering API
In order to hierarchical support test ordering, we’re going to represent ordered tests via the
GetTestFixtures will return an enumerable of types which represents the test fixtures or other
ITestCollection to run.
ContinueOnFailure will not be implemented just yet.
Implementing test ordering discovery
Implementing our test orderer is simple. We’re delegating the bulk of the work, discovering tests, to test default implementation.
The work of ordering the tests is done in a seperate class, the
OrderedTreeBuilder. It’s too much code to go into detail here, so I’Il write down the algorithm instead:
- Discover all ordered tests (
ITestCollection) in the assembly
- Recursively move ordered tests to a seperate test suite, depending on the
ITestCollectionit belongs to
- Seperate the ordered tests and unordered tests
- Sort each level of ordered tests
Implementing test ordering in our integration test
Enabling ordered testing for our integration test is now a piece of cake. In
AssemblyInfo.cs we register the ordered test assembly builder:
And we implement a
ITestCollection which contains our tests:
And we’re done!
Running the tests
To run the tests, we can simply pass our test assembly to the NUnit runner and the tests will be executed in the correct order.
Test Files BankAccountApp.NUnitTests.Integration\bin\BankAccountApp.NUnitTests.Integration.dll => BankAccountApp.NUnitTests.Integration.ChildNamespace.ChildTestFixtureWithoutOrder.Test1 => BankAccountApp.NUnitTests.Integration.ChildNamespace.ChildTestFixtureWithoutOrder.Test2 => BankAccountApp.NUnitTests.Integration.TestFixtureWithoutOrder.Test1 => BankAccountApp.NUnitTests.Integration.TestFixtureWithoutOrder.Test2 => BankAccountApp.NUnitTests.Integration.PreIntegrationTest.PreIntegrationTest_FirstStep => BankAccountApp.NUnitTests.Integration.PreIntegrationTest.PreIntegrationTest_SecondStep => BankAccountApp.NUnitTests.Integration.IntegrationTest.NewAccount_AccountRepository_CanSaveAccount => BankAccountApp.NUnitTests.Integration.IntegrationTest.ExistingAccount_AccountRepository_CanRetrieveSavedAccount => BankAccountApp.NUnitTests.Integration.IntegrationTest.ExistingAccount_AccountRepository_CanDeleteSavedAccount => BankAccountApp.NUnitTests.Integration.IntegrationTest.NonExistingAccount_AccountRepository_GetThrows => BankAccountApp.NUnitTests.Integration.PostIntegrationTest.PostIntegrationTest_FirstStep => BankAccountApp.NUnitTests.Integration.PostIntegrationTest.PostIntegrationTest_SecondStep
Resharper test runner: Resharper appears to use its own methods to discover tests in the solution, so the ordered test collections don’t show up at all. The hierarchy is not correctly displayed. It is understandable, because it would then need to load the NUnit assemblies in the Visual Studio process which cause Visual Studio to destabilize. But that’s as far as the issues go: The tests are executed in the correct order and I could confirm the ordered test assembly builder is indeed invoked by Resharper.
Visual Studio runner: The display of tests in the test explorer is a bit… sad. Currently there are only a few tests implemented, so it is probably completely unreadable when you have more than a dozen tests.
Console runner: Shows (and executes) the tests in the correct order. The output is shown earlier in this post.
NUnit GUI runner: The NUnit GUI test runner is still under construction, but already in a working state. When loading the tests in the NUnit GUI, the tree of tests, just like intented is nicely displayed. The tests are also run in the correct order.
Below are the limitations that are still present in this implementation
First limitation is that we currently don’t schedule the dependencies of a test to be run when a test is scheduled. We will look at that in a future blog post.
The second obvious limitation is that when a dependency of a test fails, the test is still executed while it should be ignored (marked as ‘not runnable’) similar to what happens when a test (fixture) setup method fails.
The third limitation, but this is only a small one, is that when you instruct NUnit to run the failed tests first, it will ignore the dependencies of that test (see also limitation two above).
We’ve seen how we can use a custom
ITestAssemblyBuilder to implement ordered testing in NUnit. The only question you might want to ask yourself is: Do I want to use an fork of NUnit to keep use this implementation? If you have a fork, you’re responsible to the users of your fork to keep it updated with the main repository of NUnit.
Also, keep in mind that there is currently being discussed of implementing test ordering in NUnit beyond test methods. Before an official implementation reaches NUnit, it may be a while though, these people put their spare time in developing open-source projects!
The code of this blog post can be found on GitHub. The code of the NUnit fork can also be found on GitHub (Diff).
What are your thoughts?