In this article, Implementing a Test Harness Using Automation Services, Andreas Stiebe presented the source code for a simple unit test framework built using TIBCO Spotfire Automation Services and NUnit 2.5.2.  In this post, I present some best practices around unit testing and demonstrate step-by-step how to use the framework to test a custom extension to TIBCO Spotfire.  Developers who routinely use unit testing as part of their own in-house development methodology are free to use the code and principles I present here to implement unit tests for their own custom Spotfire extensions.

The unit tests described in this article are available as a complete Microsoft Visual Studio project available here.

The structure of a test class

Unit tests are written using the same design patterns as NUnit, on which our test harness is based.  Tests are written inside a test class, with each test implemented as a particular method.  Setup and tear-down (cleanup) logic is also inserted into individual methods.  When our test framework calls each of these methods, it provides access to the AnalysisApplication object from the running Automation Services job. 

The unit test framework defines six custom attributes which can be used to adorn your test classes and methods:

Class attributes:

[SpotfireTestFixture] - this attribute indicates that a class is a “test fixture”, or a class containing unit tests.

Method attributes:

[SpotfireSetupFixture] - each test class can optionally contain one method adorned with this attribute.  It indicates the method contains setup logic that should be run once before all of the tests in the class are run.

[SpotfireTearDownFixture] - each test class can optionally contain one method adorned with this attribute.  It indicates the method contains tear-down logic that should be run once after all of the tests in the class are run.

[SpotfireSetup] - each test class can optionally contain one method adorned with this attribute.  It indicates the method contains setup logic that should be run before each test in the class.

[SpotfireTearDown] - each test class can optionally contain one method adorned with this attribute.  It indicates the method contains tear-down logic that should be run once after each test in the class.

[SpotfireTest] – every test method in the class should be adorned with this method.

Every method adorned with one of the above attributes should have the following signature:

public void TestOrSetupOrTeardown(AnalysisApplication application)

Best Practices of Unit Testing

• Each unit test should be atomic, meaning that each test method should test one indivisible “unit” of functionality.   For instance this could involve testing the outcome of one branch of business logic in a method, or testing the return value of the get method for a property.
• Each unit test should be independent so that they can run in any order.
• Each unit test should be self-contained, meaning they should not rely on the results or existence of other tests.

In addition to these best practices, I recommend that you write all custom tests in a separate Visual Studio project (and hence separate .NET assembly).  This provides a distinct separation between the assembly containing the business logic and the assembly containing the tests, ensuring that incomplete or obsolete test code can never prevent your Spotfire extension from compiling.  If your tests are failing to compile, you can always temporarily unload the test project and continue debugging.

This instantly provides a significant challenge – how can one test business logic contained in private class members?

There are two viable solutions for this:

1. Use only internal rather than private members, and annotate your assembly with the InternalVisibleTo attribute.  This is relatively easy to implement but does have the disadvantage of negating some of the best practices of member accessibility.
2. Use reflection to invoke the private members.  Fortunately this non-trivial (and somewhat questionable) practice is made easier and more robust by Visual Studio in the form of private accessors.

Visual Studio 2008 Professional and upward editions contain functionality for aiding and performing unit testing.  One of the capabilities is the ability to generate private accessors.  Private accessors are assemblies which contain the reflection code necessary to access private, internal and protected members in the target assembly.  We can utilise this capability without having to use the rest of Visual Studio’s unit testing functionality.  We’ll examine the steps for generating a private accessor in the next section.

An example of a unit test project

Prerequisites:

• Spotfire SDK\Examples\Extensions\SpotfireDeveloper.CustomPanelExample
  The project contains the code of a custom panel object.  We will build a test project to test the simple logic in the BasicCustomPanel and BasicCustomPanelFactory classes.
• All prerequisites shown in the Requirements section here.

First, open the CustomPanelExample project in Visual Studio and save it in a new solution.

Now add a new Spotfire extension project to the solution with a suitable name (such as SpotfireDeveloper.CustomPanelExampleTests) and add the following references:

• The Spotfire libraries (Spotfire.Dxp.Application, Spotfire.Dxp.Data and Spotfire.Dxp.Framework)
• The NUnit libraries (nunit.core, nunit.core.interfaces, nunit.framework and nunit.util)
• SpotfireDeveloper.CustomPanelExample (project reference)
• SpotfireDeveloper.TestHarness (project or binary reference)
• Microsoft.VisualStudio.QualityTools.UnitTestFramework.

This extension project will contain our test code.

Next we will temporarily generate a new Visual C# test project, which will provide the settings we need to add to our Spotfire extension project.  Unfortunately Visual Studio does not allow the initial generation of a private accessor for any project type other than a Visual C# test project.

In Visual Studio, select Test > New Test…

Choose to add a new Unit Test, and to Create a new Visual C# test project…

Click OK, supply a suitable project name and click Create.

In the SpotfireDeveloper.CustomPanelExample project, open one of the class files, right click on the text editor in Visual Studio and select Create Private Accessor:

This will now have created the private accessor in the Visual C# test project.  You will notice that Visual Studio has created a new folder in the project called Test References.  Save your solution and close Visual Studio.  Now, in a text editor, edit the .csproj files from both the Visual C# test project and our Spotfire extension test project.  The relevant piece of XML pertaining to the private accessor in the Visual C# test project should look similar to the following:

  <ItemGroup>
    <None Include="SpotfireDeveloper.snk" />
    <Shadow Include="Test References\SpotfireDeveloper.CustomPanelExample.accessor">
    </Shadow>
  </ItemGroup>

Copy this section into the corresponding place in the .csproj file of our Spotfire test extension.  Next, copy the entire Test References folder from the root of the Visual C# test project into the Spotfire test extension project.

Upon reopening Visual Studio, the Spotfire test extension project should now show the test references folder and the private accessor as shown below:

The Visual C# test project and the other content added by the Visual Studio unit test framework (such as the “Solution Items” added at the very top of the solution) can now be removed from the project and deleted.  For more information around private accessors, see Create and Run a Unit Test for a Private Method and Unit Tests for Private, Internal, and Friend Methods.

Now we are ready to start adding some unit tests.  Create a new class to contain the unit tests, called BasicCustomPanelTests.cs.

Add the following using statements, and add the attribute stating that this class is a text fixture:

using System;
using System.Collections.Generic;
using System.Text;

using MSUnit = Microsoft.VisualStudio.TestTools.UnitTesting;

using NUnit.Framework;

using Spotfire.Dxp.Application;
using Spotfire.Dxp.Application.Visuals;
using Spotfire.Dxp.Framework.DocumentModel;
using Spotfire.Dxp.Framework.Library;

using SpotfireDeveloper.CustomPanelExample;
using SpotfireDeveloper.TestHarness;

namespace SpotfireDeveloper.CustomPanelExampleTests
{
    [SpotfireTestFixture]
    public class BasicCustomPanelTests
    {
    }
}

Next we are going to add some private fields to the class and populate these in some setup/teardown code.  Our tests will use a particular data set – in this case I have used the /Demo/Analysis Files/Baseball/Baseball file that is shipped with the Spotfire library, and have saved this to the location /Test Harness/BasicCustomPanelTestData in the library.

Add the following private fields to the class:

private Page page1;
private Page page2;
private BasicCustomPanel panel1;
private BasicCustomPanel panel2;

private bool pageRenamed;
private bool visualItemsChanged;
private int pageInfoTriggerCalls;

Add the following fixture-level setup code – this code will open a particular analysis file from the library:

[SpotfireSetupFixture]
public void SetupFixture(AnalysisApplication application)
{
    LibraryManager manager = application.GetService<LibraryManager>();
    LibraryItem item = null;

    if (manager.TryGetItem("/Test Harness/BasicCustomPanelTestData",
            LibraryItemType.Analysis,
            out item))
    {
        application.Open(item, new DocumentOpenSettings());
    }
    else
    {
        Assert.Fail("SetupFixture failed - unable to find library item.");
    }

    // Remove any pages in the document
    TearDownTest(application);

}

Now add the test-level setup and teardown code – this code creates two pages containing some visuals and a BasicCustomPanel.  The teardown code deletes all pages in the document:

[SpotfireSetup]
public void SetupTest(AnalysisApplication application)
{
    application.Document.Transactions.ExecuteTransaction(
        delegate
        {
            this.page1 = application.Document.Pages.AddNew();
            this.page2 = application.Document.Pages.AddNew(); 

            this.page1.Title = "Test 1";
            this.page1.Visuals.AddNew(VisualTypeIdentifiers.ScatterPlot);
            this.page1.Visuals.AddNew(VisualTypeIdentifiers.BarChart); 

            this.page2.Title = "Test 2";
            this.page2.Visuals.AddNew(VisualTypeIdentifiers.Treemap); 

            this.panel1 = page1.Panels.AddNew(CustomPanelIdentifiers.SimplePanel)    
                as BasicCustomPanel; 

            this.panel2 = page2.Panels.AddNew(CustomPanelIdentifiers.SimplePanel)
                as BasicCustomPanel;
        });
} 

[SpotfireTearDown]
public void TearDownTest(AnalysisApplication application)
{
    application.Document.Transactions.ExecuteTransaction(
        delegate
        {
            foreach (Page page in application.Document.Pages)
            {
                application.Document.Pages.Remove(page);
            }
        });
} 

Now we will add the tests.  The first test is a simple test on the getter method of the BasicCustomPanel’s runtime property.  Note how we use standard NUnit functionality such as Assert to actually implement the tests:

[SpotfireTest]
public void PageInfo_Get_Test(AnalysisApplication application)
{
    Assert.That(panel1.PageInfo,
        Is.EqualTo("This page is called 'Test 1' and has 2 visualization(s)."));

    Assert.That(panel2.PageInfo,
        Is.EqualTo("This page is called 'Test 2' and has 1 visualization(s)."));
}

The second test does the same thing, so is redundant, but it is an example of how to use the private accessor to directly call the static methods used to calculate the values of the runtime property:

[SpotfireTest]
public void PageInfoComputeMethod_Test(AnalysisApplication application)
{
    Assert.That(BasicCustomPanel_Accessor.PageInfoComputeMethod(panel1),
        Is.EqualTo("This page is called 'Test 1' and has 2 visualization(s)."));

    Assert.That(BasicCustomPanel_Accessor.PageInfoComputeMethod(panel2),
        Is.EqualTo("This page is called 'Test 2' and has 1 visualization(s)."));
}

The next test creates an ExternalEventHandler and hooks onto the runtime property.  It then changes the properties on which the runtime property depends and validates that the events fired successfully:

[SpotfireTest]
public void PageInfo_Trigger_Test(AnalysisApplication application)
{
    this.pageInfoTriggerCalls = 0;

    // Be sure to read the runtime property first or it will never invalidate.
    string a = panel1.PageInfo;
    a = panel2.PageInfo;

    using (ExternalEventManager manager1 = new ExternalEventManager(),
        manager2 = new ExternalEventManager())
    {
        manager1.AddEventHandler(PageInfoChangedHandler,
            Trigger.CreatePropertyTrigger(panel1,
                BasicCustomPanel.PropertyNames.PageInfo));
        manager2.AddEventHandler(PageInfoChangedHandler,
            Trigger.CreatePropertyTrigger(panel2,
                BasicCustomPanel.PropertyNames.PageInfo));

        page1.Transactions.ExecuteTransaction(
            delegate
            {
                page1.Visuals.AddNew(VisualTypeIdentifiers.BarChart);
            });

        page1.Transactions.ExecuteTransaction(
            delegate
            {
                page2.Title = "A new title";
            });

        Assert.That(this.pageInfoTriggerCalls, Is.EqualTo(2));
    }

}

private void PageInfoChangedHandler()
{
    this.pageInfoTriggerCalls++;
}

This test is again redundant, but it directly tests the Triggers setup by the static methods which define the runtime property.  Again it provides an example of using the private accessor:

[SpotfireTest]
public void PageInfoDependencyDeclarer_Test(AnalysisApplication application)
{
    this.pageRenamed = false;
    this.visualItemsChanged = false;

    using (ExternalEventManager manager1 = new ExternalEventManager(),
        manager2 = new ExternalEventManager())
    {
        manager1.AddEventHandler(PageRenamedHandler, BasicCustomPanel_Accessor.PageInfoDependencyDeclarer(panel1));
        manager2.AddEventHandler(VisualItemsChangedHandler, BasicCustomPanel_Accessor.PageInfoDependencyDeclarer(panel2));

        page1.Transactions.ExecuteTransaction(
            delegate
            {
                page1.Title = "A new title";
                page2.Visuals.AddNew(VisualTypeIdentifiers.BarChart);
            });

        Assert.That(this.pageRenamed, Is.True);
        Assert.That(this.visualItemsChanged, Is.True);
    }

}

private void PageRenamedHandler()
{
    this.pageRenamed = true;
}

private void VisualItemsChangedHandler()
{
    this.visualItemsChanged = true;
}

The final test tests the BasicCustomPanelFactory class.  It checks that the CreateCore method correctly returns a BasicCustomPanel and is an example of how to use the private accessor to call a non-public instance method on an object:

[SpotfireTest]
public void BasicCustomPanelFactoryTests(AnalysisApplication application)
{
    BasicCustomPanelFactory factory = new BasicCustomPanelFactory();
    MSUnit.PrivateObject wrapper = new MSUnit.PrivateObject(factory);
    BasicCustomPanelFactory_Accessor accessor = new BasicCustomPanelFactory_Accessor(wrapper);

    BasicCustomPanel customPanel = accessor.CreateCore(application);
    Assert.IsNotNull(customPanel);
}

Wrapping up

To make the tests available to the test harness, there are a few more classes which need to be created.  First of all, our set of tests must have a unique TypeIdentifier:

public sealed class ExampleTestsTypeIdentifiers : CustomTypeIdentifiers
{
    public static readonly CustomTypeIdentifier TestIdentifier =
        CreateTypeIdentifier("SpotfireDeveloper.CustomPanelExampleTests",
            "BasicCustomPanel Tests",
            "Example tests for the BasicCustomPanel extension.");
}

Next, we need to create a new TestAssemblyMarker.  The TestAssemblyMarker is used to return as Assembly object.  This is passed to the NUnit framework, which uses reflection to extract all test fixtures contained in the assembly.

using System;
using System.Reflection;

using SpotfireDeveloper.TestHarness.Extension;

namespace SpotfireDeveloper.CustomPanelExampleTests
{
    public class ExampleTestsAssemblyMarker : CustomTestAssemblyMarker
    {
        public ExampleTestsAssemblyMarker()
            : base(ExampleTestsTypeIdentifiers.TestIdentifier)
        {
        }

        public override System.Reflection.Assembly GetTestAssembly()
        {
            return Assembly.GetAssembly(typeof(ExampleTestsAssemblyMarker));
        }
    }
}

Finally we need to create an AddIn class, of the type TestFrameworkAddIn declared by the unit test framework, to add our unit tests into Spotfire:

using System;
using System.Collections.Generic;
using System.Text;

using SpotfireDeveloper.TestHarness.Extension;

namespace SpotfireDeveloper.CustomPanelExampleTests
{
    public class ExampleTestsAddIn : TestFrameworkAddIn
    {
        protected override void RegisterTests(TestFrameworkRegistrar registrar)
        {
            registrar.Register(new ExampleTestsAssemblyMarker());
        }
    }
}

Once all this has been built in Visual Studio, the relevant projects need to be added into the Spotfire Package Builder:

Now the project can either be run from the Package Builder by clicking Run Configuration, or debugged from Visual Studio (as per standard practice, you can use Spotfire SDK\Starter\Spotfire.Dxp.exe as the external program for debugging).

In the Automation Services Job Builder, you should now see BasicCustomPanel Tests as an available test assembly:

In the source code I have provided at the top of this article, you will find a sample Automation Services job file called TestJob.xml.  This utilises the Sample Test Results.dxp result template provided by Andreas in the source code in his article:

The job will run the tests in the test project, save the updated result template into the Spotfire library and then send an email containing a summary of the test results.  A sample email is shown here:

If you publish your test project DLL to the Automation Services server as described in Andreas’ article, you can automate the running of the tests against the server.  Personally, I run a local installation of Automation Services explicitly to allow automated testing.  Having installed Automation Services locally, you can publish the DLLs from your project using a build event in Visual Studio:

xcopy "$(TargetDir)SpotfireDeveloper.CustomPanelExample.dll" "C:\Program Files (x86)\TIBCO\Spotfire\AutomationServices\bin" /Y
xcopy "$(TargetDir)SpotfireDeveloper.CustomPanelExample_Accessor.dll" "C:\Program Files (x86)\TIBCO\Spotfire\AutomationServices\bin" /Y
xcopy "$(TargetPath)" "C:\Program Files (x86)\TIBCO\Spotfire\AutomationServices\bin" /Y

Note that you will typically need Administrator privileges to write to the bin directory of Automation Services (in Windows Vista or Windows 7 you will need to explicitly run Visual Studio as Administrator if UAC is enabled).

Finally, in this scenario you can write a simple batch file to trigger off the automatic test harness:

Spotfire.Dxp.Automation.ClientJobSender.exe http://localhost:85/AutomationServices/JobExecutor.asmx TestJob.xml

Whenever you build your main Spotfire extension, you only then need to double click on the batch file to trigger the test harness.  Shortly afterwards you will then receive an email in your inbox with the results of the tests.

Conclusion

Hopefully this article has provided you with enough information to start using the Automation Services test harness to incorporate unit test into your custom Spotfire extensions.  Also it has provided a few best practices on writing unit tests and on how to structure your unit test projects to minimise the impact on the actual extension code itself.