Boost Test Adapter Design

Boost Unit Test Adapter Design

The Boost Unit Test Adapter allows users to discover and execute Boost Test test cases from within the Visual Studio IDE. The adapter abides to Visual Studio's Microsoft.VisualStudio.TestPlatform.ObjectModel specifications which, in brief, expect an implementation to list tests available within a compiled module and be able to execute tests individually given either a module or a test case specification as input. For more information regarding Visual Studio's test platform, refer to the Visual Studio Test Platform Primer article.

The following is a high-level description of how the Boost Unit Test Adapter handles test discovery and execution for native C++ modules containing Boost Tests.

Test Discovery

Once an exe or dll is successfully compiled, the Boost Unit Test Adapter needs to verify if the module contains Boost Tests. Identifying test cases from C++ native modules has its fair share of challenges. In any case, test discovery should not be limited to one approach; multiple approaches should be available and one should fall-back to the other in case an approach fails to find tests.

An appropriate discovery strategy is first identified for a given set of modules and later, these strategies are applied.

Currently, there are 3 main approaches to discovery (hereby referred simply as discoverers) defined.

• Source Code Discoverer

The original discoverer which identifies Boost Test cases from the module's source code. The discoverer (tries to) attaches to the current Visual Studio running instance and locates all projects which generate the modules specified as input. If a project is successfully mapped to module, the project is later flagged for inspection. Once all projects are identified, each project's sources are parsed in an attempt to locate automatically registered Boost test cases.

• List Content Discoverer

Boost Test 3 test runner allows test cases to be listed. A module is first tested to see whether this functionality is available and if so, the module is registered with this discovery approach. Test listing is parsed and symbol information is used to identify source information for test cases.

• External Discoverer

Via test adapter configuration, users can specify an external Boost Test runner which should be responsible of listing and executing tests. If an external discoverer is configured, this takes precedence over the 'internal' ones. External test runners should explicitly list their test content in a pre-determined format either generated via an external process or embedded within the test adapter configuration.

Any and all discovered tests by any of the mentioned approach is identified to the Visual Studio API's. Test suite information is also provided to Visual Studio so that it is possible to categorize test cases via test suites.

Test Execution

Once tests are discovered from a module, the Boost Test Adapter attempts to execute test cases. Each test case is individually executed in a separate process so that test results can be communicated immediately back to Visual Studio. Results generated by the process are later parsed and sent to Visual Studio.

• Code Coverage

Executing test cases individually can be slow, especially for code coverage since symbols need to be reloaded per process spawn. In an attempt to minimize this cost, tests are batched per test suite. Test cases contained within the same test suite are then executed in one go.