How We Test

Testing IdentityModel

Goals of our testing

1. Multiple variations for similar scenarios:
   • Asymmetric signatures need to be tested for a number of algorithms and key types
2. Ability to easily break on a specific variation.
3. Ensure objects are as expected when created from different call graphs.
   • When validating a token, we need to ensure that the identity presented to the application is deterministic and consistent.
4. Minimize breaking changes:
   • Ensure Exceptions thrown are the same between releases.
   • Extensibility (virtuals, delegates) are called consistently.
   • Naming and ordering of parameters does not change.
5. Minimize unit test time
   • Test run time is limited to 20 minutes on builds; it will fail and timeout after this set amount of time has passed.

Testing Framework

We are using XUnit since it is easy to use, integrates with VisualStudio and has the ability to run multiple variations. We focus a lot of testing using TheoryData. You can find XUnit here: https://github.com/xunit

Test Model

Main components of our model are:

1. TheoryDataBase is a derived TheoryData type from XUnit that has some specific properties. Normally a derived TheoryDataBase is designed for a specific test.
2. CompareContext is used to collect issues and report in a test run.
3. IdentityComparer is used to detect if two objects are equal. Can populate a CompareContext.
4. ExpectedException is used to check that exceptions are of the correct type and have the expected message. Adding a postfix of ':' when specifying the expected exception message helps ensure correctness.

We have a class TestStubTests that is a good place to understand the layout. The stub looks like:

public class TestStubTests
{
    [Theory, MemberData(nameof(TestStubTheoryData))]
    public void TestStubTest1(TestStubTheoryData theoryData)
    {
        var context = TestUtilities.WriteHeader($"{this}.TestStubTest1", theoryData);
        try
        {
            var obj = new object();
            theoryData.ExpectedException.ProcessNoException(context);
            if (theoryData.CompareTo != null)
                IdentityComparer.AreEqual(obj, theoryData.CompareTo, context);
        }
        catch (Exception ex)
        {
            theoryData.ExpectedException.ProcessException(ex, context);
        }

        TestUtilities.AssertFailIfErrors(context);
    }

    public static TheoryData<TestStubTheoryData> TestStubTheoryData
    {
        get
        {
            return new TheoryData<TestStubTheoryData>
            {
                new TestStubTheoryData
                {
                    First = true,
                    ExpectedException = ExpectedException.SecurityTokenInvalidLifetimeException("IDX10230:"),
                    TestId = "TestStub1"
                }
            };
        }
    }
}

public class TestStubTheoryData : TheoryDataBase
{
    public object CompareTo { get; set; }
}

Key Details

1. Obtain a CompareContext calling TestUtilities.WriteHeader. This will print a nice string for the test run and assist with debugging when it fails.
2. In a try - catch call ExpectedException.ProcessNoException - ProcessException, passing CompareContext.
3. Use IdentityComparer to check expected values, passing CompareContext.
4. Finish by calling TestUtilities.AssertFailIfErrors, passing CompareContext.
5. When setting TheoryDataBase.TestId, do not use spaces, underscores, dashes, periods. This is so that within the VisualStudio test explorer window, you can double click the TestId and copy to the clipboard.
6. Always have the test take a single parameter derived from TheoryDataBase, that way a conditional break point can be set to: theoryData.TestId == "copied from clipboard".