Ordered testing with XUnit, NUnit and MSTest part 4: XUnit
In the previous post we looked at ordered testing in NUnit. Today we are going to implement ordered tests in XUnit. The code for this post can be found on GitHub. Disclaimer: This code will have rough edges, and may not work for you, kill you cat or blow up in your face.
This post is part of a series of posts on ordered testing:
- Ordered testing with XUnit, NUnit and MSTest part 1: The setup
- Ordered testing with XUnit, NUnit and MSTest part 2: MSTest
- Ordered testing with XUnit, NUnit and MSTest part 3: NUnit
- Ordered testing with XUnit, NUnit and MSTest part 4: XUnit
- Ordered testing with XUnit, NUnit and MSTest part 5: NUnit implementation revised
- Ordered testing with XUnit, NUnit and MSTest part 6: NUnit implementation revised part 2
XUnit
XUnit doesn’t have built-in ordering, but does appear to have the interfaces to support it: ITestCaseOrderer
for ordering methods within a class, and ITestCollectionOrderer
for ordering test collections. There doesn’t appear to be a interface for class fixtures, so we’Il have to work with these interfaces.
The interfaces can be registered by placing an TestCaseOrdererAttribute
or TestCollectionOrdererAttribute
respectively on the assembly. The signatures of both interfaces are fairly straightforward: Take an enumerable and return an ordered enumerable. Let’s go and implement these.
Setting up the boilerplate
First we set up some general attributes that actually allow us to specify dependencies. We define an attribute for methods:
And we define a method for test collections:
Test dependency discovery
XUnit finds the test cases or test collections and passes them through our interface. We will re-use the test sorting code from the previous post so we’Il just implement a wrapper that allows our existing code to talk to the test cases:
We do the same for test collections, but we need also a lookup between type and collection name so we can easily look up the type bij collection name.
Now it is just a matter of registration of the implementations in the test assembly, and we’re done!
Running the tests
To run the tests, we can simply pass our test assembly to the XUnit runner and the tests will be executed in the correct order.
Discovering: BankAccountApp.XUnitTests.Integration
Discovered: BankAccountApp.XUnitTests.Integration
Starting: BankAccountApp.XUnitTests.Integration
BankAccountApp.XUnitTests.Integration.PreIntegrationTest.PreIntegrationTest_FirstStep
BankAccountApp.XUnitTests.Integration.PreIntegrationTest.PreIntegrationTest_SecondStep
BankAccountApp.XUnitTests.Integration.IntegrationTest.NewAccount_AccountRepository_CanSaveAccount
BankAccountApp.XUnitTests.Integration.IntegrationTest.ExistingAccount_AccountRepository_CanRetrieveSavedAccount
BankAccountApp.XUnitTests.Integration.IntegrationTest.ExistingAccount_AccountRepository_CanDeleteSavedAccount
BankAccountApp.XUnitTests.Integration.IntegrationTest.NonExistingAccount_AccountRepository_GetThrows
BankAccountApp.XUnitTests.Integration.PostIntegrationTest.PostIntegrationTest_FirstStep
BankAccountApp.XUnitTests.Integration.PostIntegrationTest.PostIntegrationTest_SecondStep
Finished: BankAccountApp.XUnitTests.Integration
=== TEST EXECUTION SUMMARY ===
BankAccountApp.XUnitTests.Integration Total: 8, Errors: 0, Failed: 0, Skipped: 0, Time: 0,389s
Obvious advantage over MSTest is that this doesn’t need any special configuration when calling the test runner. It “just works”.
Limitations
There are three limitations in the current implementation:
Each test class needs to be put in an unique test collection, but then test collections can be ordered against each other. However, if you are already using test collections and want to order classes against each other, this solution will not work.
Within a test collection, tests can be ordered. When a test fails, the other tests are happily scheduled for execution and not ignored.
If one test is selected for execution, the dependencies of that test should also be executed. Currently only the test itself is executed.
Tooling support
Visual Studio runner: The tests are not correctly ordered in the Test Explorer, but the tests themselves execute in the correct order. Grouping tests makes no difference. The test explorer probably applies its own ordering.
Resharper runner: While the test tree of Resharper does not appear to follow the correct ordering, the tests are actually performed in the correct order. Resharper probably uses its own mechanism to discover the tests.
Console runner: Shows (and executes) the tests in the correct order.
Conclusion
We have looked at implementing ordered testing in XUnit. Given you don’t use test collections at the moment this method may be useful to you. Happy testing!
The code for this post can be found on GitHub.
What are your thoughts?