Modules.md

.NET Core Modules

Modules on the dotnet platform are .NET Core assemblies that export one or more implementations of different instantiations of the generic ITest<TInput, TOutput> interface. The interface is defined in the DotnetRunner assembly, and the exporting is done using MEF.

Adding new modules

1. Create a new .NET Core class library project /src/dotnet/modules/YourModule/YourModule.XXproj. The extension of the project file name depends on the chosen language. Any language that can be compiled into a .NET Core class library can be used. This file structure is expected by the global runner.

2. In your project add a reference to DotnetRunner:

   <ItemGroup>
       <ProjectReference Include="..\..\runner\DotnetRunner.csproj" />
   </ItemGroup>

3. Create /src/dotnet/modules/YourModule/CMakeLists.txt with the following content where YourModule is the name of your module:

   project("ADBench_YourModule")
   
   add_custom_target(YourModule ALL
       COMMAND "dotnet" "publish" "--configuration" "${AD_BUILD_TYPE}" "--output" "${CMAKE_CURRENT_BINARY_DIR}" "-p:IntermediateOutputPath=${CMAKE_CURRENT_BINARY_DIR}/obj/" "${CMAKE_CURRENT_LIST_DIR}/YourModule.XXproj"
       )

4. Add the following line to /src/dotnet/modules/CMakeLists.txt:

   add_subdirectory ("YourModule")

   These two steps ensure that build environment generated by cmake will build your module as well.

5. For each objective create a class implementing the interface ITest<TInput, TOutput> where T is an objective type (GMM, BA etc.). You can find this interface in /src/dotnet/runner/ITest.cs. TInput and TOutput are input and output types specific for the objective there. They are defined in corresponding /src/dotnet/runner/Data/TData.cs files.

   The functions you need to implement:

   • void Prepare(TInput input)

     Converts the input data from the TInput type in which it is provided by the calling benchmark runner into the format optimized for use with the tested AD framework. Stores it.

     Optionally, performs other preparatory activities need by the tested AD framework.

   • void CalculateObjective(int times)

     Repeatedly computes the objective function times times for the stored input. Stores results.

   • void CalculateJacobian(int times)

     Repeatedly computes the Jacobian of the objective function times times for the stored input. Stores results.

   • TOutput Output()

     Converts internally saved outputs into the format specified by the runner.

6. Decorate each such class with an Export attribute:

   [Export(typeof<DotnetRunner.ITest<TInput, TOutput>>)]

   where T is still an objective type here.

   This will allow the runner to load these classes using MEF.

7. Add your module to common dotnet module tests project (see below).

8. Add your module to the global runner script.

Unit Tests

We use xUnit framework to test .NET Core assemblies.

Adding tests for a new module

AD Bench already contains some tests for each objective. When you add a new module, the first thing you should do is to test your module with the existing tests. Follow these steps for every objective you want to test:

1. Add a reference to your project in /test/dotnet/modules/DotnetModulesTests.csproj

   <ProjectReference Include="..\..\..\src\dotnet\modules\DiffSharpModule\DiffSharpModule.fsproj">
     <Private>true</Private>
   </ProjectReference>

   Note the Private flag - it's needed to ensure, that your module will be copied locally for the test project, which is needed to test the dynamic loading.

2. Open /test/dotnet/modules/TTests.cs where T is the short name of the tested objective. You will see the following lines:

   public static IEnumerable<object[]> TestParameterSet { get; } = new[]
   {
       new ModuleTestParameters("./DiffSharpModule.dll", 1e-10 ),
       ...
   }.Select(m => new object[] { m }).ToArray();

3. Add a ModuleTestParameters object for your module to the list. Arguments of the constructor here are the name of the assembly with your module and a tolerance for the test results.

   public static IEnumerable<object[]> TestParameterSet { get; } = new[]
   {
       new ModuleTestParameters("./DiffSharpModule.dll", 1e-10 ),
       ...
       new ModuleTestParameters("./YourModule.dll", absoluteTolerance )
   }.Select(m => new object[] { m }).ToArray();

   tolerance is a number used to compare results produced by the current module with the correct results. If an absolute difference between at least one of them exceeds this value then the test is failed.

Adding a new common test case

Follow these steps to add a new test case for an existing objective that will be shared by all modules:

1. Open /test/dotnet/modules/TTests.cs where T is the short name of the testing objective.
2. Add the following lines to the end of the TTests class:

       [Theory]
       [MemberData(nameof(TestParameterSet))]
       public void TestCaseName(ModuleTestParameters testParameters)
       {
           using (var moduleLoader = new ModuleLoader(testParameters.ModuleName))
           {
               var test = moduleLoader.GetTTest();
               ...
               test code
               ...
           }
       }

   There you can test the behavior of the ITest<TInput, TOutput> test object. The test will run for all modules listed in TestParameterSet.

Adding new objectives

To create a common test suite for a new objective create /test/dotnet/modules/TTests.cs with the following content:

using DotnetRunner;
using DotnetRunner.Data;
using System.Collections.Generic;
using System.Linq;
using Xunit;

namespace DotnetModulesTests
{
    public class TTests
    {
        public static IEnumerable<object[]> TestParameterSet { get; } = new[]
        {
            new ModuleTestParameters("./ModuleName1.dll", 1e-10 ),
            new ModuleTestParameters("./ModuleName2.dll", 1e-10 ),
            ...
        }.Select(m => new object[] { m }).ToArray();
    }
}

In the TestParameterSet list the modules that support your objective and respective allowed tolerances. Create test cases as described in the "Adding a new common test case" section.