In the previous post we looked briefly at ordered testing in MSTest. Today we are going to implement ordered tests in NUnit. The code for this post can be found on GitHub. Disclaimer: This code will have rough edges, and may not work for you, kill you cat or blow up in your face. We will refine the code in a later post.
This post is part of a series of posts on ordered testing:
- Ordered testing with XUnit, NUnit and MSTest part 1: The setup
- Ordered testing with XUnit, NUnit and MSTest part 2: MSTest
- Ordered testing with XUnit, NUnit and MSTest part 3: NUnit
- Ordered testing with XUnit, NUnit and MSTest part 4: XUnit
- Ordered testing with XUnit, NUnit and MSTest part 5: NUnit implementation revised
- Ordered testing with XUnit, NUnit and MSTest part 6: NUnit implementation revised part 2
NUnit has limited built-in support for test ordering, and no support for tests with dependencies on other tests. Since NUnit 3.2 there is support for ordering individual tests in a test fixture, but it is not possible to order test fixtures. Let’s start by looking how the current way is implemented. Test method ordering is implemented in NUnit using the
OrderAttribute. If you look at the implementation of this attribute, you’Il find it implements
IApplyToTest. This is an interface that allows an attribute to modify the test. Also note that this attribute may only be applied to methods, not to classes.
When a test is discovered in NUnit and the test is an assembly, test fixture or test method, all the attributes on that test are iterated. Each attribute which implements
IApplyToTest is invoked.
In this case we find:
The property which is added to the test is picked up in
CompositeWorkItem.CreateChildWorkItems. Tests with no order are sorted last, tests with order are sorted according to the
Order property in the dynamic properties bag.
Setting up the boilerplate
First we set up some general attributes that actually allow us to specify dependencies. We define attributes for a method to specify its dependency:
And we define attributes for test fixtures, but these accept a
Test dependency discovery
Now we define a component to allow us to find and store the current test methods in the assembly.
We define an interface that will be implemented by both our representations of test methods and test fixtures:
And finally our representation of a test method:
Now we implement the same set of classes for test fixtures.
LinkedList<T> we can easily sort the tests based on their given dependencies. Each node in the linked list is (as you can guess from the name) linked to a next or previous node. For each test we add the test to the list at the start, or find the dependency and link it there.
Running the tests
To run the tests, we can simply pass our test assembly to the NUnit runner and the tests will be executed in the correct order.
Test Files BankAccountApp.NUnitTests.Integration\bin\BankAccountApp.NUnitTests.Integration.dll => BankAccountApp.NUnitTests.Integration.PreIntegrationTest.PreIntegrationTest_FirstStep => BankAccountApp.NUnitTests.Integration.PreIntegrationTest.PreIntegrationTest_SecondStep => BankAccountApp.NUnitTests.Integration.IntegrationTest.NewAccount_AccountRepository_CanSaveAccount => BankAccountApp.NUnitTests.Integration.IntegrationTest.ExistingAccount_AccountRepository_CanRetrieveSavedAccount => BankAccountApp.NUnitTests.Integration.IntegrationTest.ExistingAccount_AccountRepository_CanDeleteSavedAccount => BankAccountApp.NUnitTests.Integration.IntegrationTest.NonExistingAccount_AccountRepository_GetThrows => BankAccountApp.NUnitTests.Integration.PostIntegrationTest.PostIntegrationTest_FirstStep => BankAccountApp.NUnitTests.Integration.PostIntegrationTest.PostIntegrationTest_SecondStep Test Run Summary Overall result: Passed Test Count: 8, Passed: 8, Failed: 0, Inconclusive: 0, Skipped: 0
Obvious advantage over MSTest is that this doesn’t need any special configuration when calling the test runner. It “just works”.
When NUnit groups by test fixtures by namespace. That grouping is also considered to be a test by NUnit. This means that when NUnit sorts the tests, it will only sort the tests on the “current level” which may be a fixture or a namespace. In a future post we will reimplement test ordering so it works across namespaces.
Another limitation is that we currently don’t schedule the dependencies of a test to be run when a test is scheduled. We will look at that in a future blog post.
The third obvious limitation is that when a dependency of a test fails, the test is still executed while it should be ignored (marked as ‘not runnable’) similar to what happens when a test (fixture) setup method fails.
The fourth limitation, but this is only a small one, is that when you instruct NUnit to run the failed tests first, it will ignore the dependencies of that test (see also limitation two above).
How does the tooling support ordered tests?
Visual Studio runner: The tests are not correctly ordered in the Test Explorer, but the tests themselves execute in the correct order. When grouping tests by trait, the test fixtures are shown in the correct order but the underlying tests are not. This is not suprising because the test isn’t ordered when discovered, but only when executed.
Resharper runner: While the test tree of Resharper does not appear to follow the correct ordering, the tests are actually performed in the correct order. This is not suprising because the test isn’t ordered when discovered, but only when executed.
Console runner: Shows (and executes) the tests in the correct order. The output is shown earlier in this post.
We have looked at implementing ordered testing in NUnit. You probably don’t place all your test fixture in the same namespace, so this implementation isn’t very useful. In a later post we will look at working around these limitations.
The code for this post can be found on GitHub.
In the next post we will take a look at implementing test ordering in XUnit.