This post is part of a series of posts on ordered testing:
- Ordered testing with XUnit, NUnit and MSTest part 1: The setup
- Ordered testing with XUnit, NUnit and MSTest part 2: MSTest
- Ordered testing with XUnit, NUnit and MSTest part 3: NUnit
- Ordered testing with XUnit, NUnit and MSTest part 4: XUnit
- Ordered testing with XUnit, NUnit and MSTest part 5: NUnit implementation revised
- Ordered testing with XUnit, NUnit and MSTest part 6: NUnit implementation revised part 2
MSTest is the only framework of the three frameworks which has built-in support for ordered tests. Ordered tests are defined in an
.orderedtest file, which is an XML file containing references to each test. This file can be created easily from the Visual Studio IDE using the visual editor, as shown below. You can specify that an ordered test should be aborted if one of the tests fail. This is useful if the tests have a functional dependency on one another. If the tests don’t have a functional dependency on one another but one test mustn’t execute before the other you can leave the option disabled.
The editor generates the XML file as shown below.
The XML file is hard to edit by hand, because every test has a GUID-like as identifier. The identifier is composed like this:
- Take the name of the test method prefixed with the full name of the test class. For example:
- Convert the string to bytes (using
- Run the byte array through the
- Copy the first 128 bits of the hash (16 bytes) to a new array
- Initialize a
System.Guidbased on the new array
Note that the XML defines the relative path to the assembly in which the test method is implemented. As far as MSTest is concerned, the
.orderedtest file is a test of its own. You can pass the path to the
.orderedtest file to the Visual Studio test runner
vstest.console.exe and it will execute the ordered test file.
The biggest disadvantages of ordered tests in MSTest is that the tests themselves don’t have a hard dependency on the ordering, so you can execute an test individually which may not work because it is dependend on state from another test.
Running the tests
To run the composed ordered test, just give the Visual Studio test runner the path to the ordered test file. Don’t give the test runner the assembly file, because then it will run the tests in whatever order it likes.
Microsoft (R) Test Execution Command Line Tool Version 14.0.25123.0 Copyright (c) Microsoft Corporation. All rights reserved. Starting test execution, please wait... Passed 1- PreIntegrationTest_FirstStep (Integration) Passed 2- PreIntegrationTest_SecondStep (Integration) Passed 3- NewAccount_AccountRepository_CanSaveAccount (Integration) Passed 4- ExistingAccount_AccountRepository_CanRetrieveSavedAccount (Integration) Passed 5- ExistingAccount_AccountRepository_CanDeleteSavedAccount (Integration) Passed 6- NonExistingAccount_AccountRepository_GetThrows (Integration) Passed 7- PostIntegrationTest_FirstStep (Integration) Passed 8- PostIntegrationTest_SecondStep (Integration) Total tests: 8. Passed: 8. Failed: 0. Skipped: 0. Test Run Successful.
How does the tooling support ordered tests?
Visual Studio runner: Since the Test Explorer is unable to show hierarchy, the ‘child’ tests of an ordered tests are not shown. Also, each test method is shown individually in the Test Explorer. When executing tests, the tests are correctly shown in the results pane at the bottom of the Test Explorer.
Resharper runner: Does not support ordered tests, so no ordered tests are shown. Instead, all test methods are shown in hierarchy and no execution correct order is maintained, as expected.
Console runner: Shows (and executes) the tests in the correct order. Need to make sure the
.orderedtest file is given on the command line. Output is shown in earlier in this post.
In the next post we will look at NUnit.