-
Notifications
You must be signed in to change notification settings - Fork 34
Boost Test Adapter Design
The Boost Unit Test Adapter allows users to discover and execute Boost Test test cases from within the Visual Studio IDE. The adapter abides to Visual Studio's Microsoft.VisualStudio.TestPlatform.ObjectModel specifications which, in brief, expect an implementation to list tests available within a compiled module and be able to execute tests individually given either a module or a test case specification as input. For more information regarding Visual Studio's test platform, refer to the Visual Studio Test Platform Primer article.
The following is a high-level description of how the Boost Unit Test Adapter handles test discovery and execution for native C++ modules containing Boost Tests.
Once an exe or dll is successfully compiled and filtered by path, the Boost Unit Test Adapter needs to verify if the module contains Boost Tests. Identifying test cases from C++ native modules has its fair share of challenges. In any case, test discovery should not be limited to one approach; multiple approaches should be available and one should fall-back to the other in case an approach fails to find tests.
An appropriate discovery strategy is first identified for a given set of modules and later, these strategies are applied.
Currently, there are 3 main approaches to discovery (hereby referred simply as discoverers) defined.
The original discoverer which identifies Boost Test cases from the module's source code. The discoverer (tries to) attaches to the current Visual Studio running instance and locates all projects which generate the modules specified as input. If a project is successfully mapped to module, the project is later flagged for inspection. Once all projects are identified, each project's sources are parsed in an attempt to locate automatically registered Boost test cases.
Boost Test 3 test runner allows test cases to be listed. A module is first tested to see whether this functionality is available and if so, the module is registered with this discovery approach. Only modules which are deemed as safe, that is modules which contain the --list_content debug symbol, are taken into consideration but this behaviour can be overriden via the <ForceListContent> configuration value which is ideal for cases where BOOST Test is dynamically linked. DOT test listings are then parsed via an ANTLR4 generated grammar and later identified to the Visual Studio APIs.
Via test adapter configuration, users can specify an external Boost Test runner which should be responsible of listing and executing tests. If an external discoverer is configured, this takes precedence over the 'internal' ones. External test runners should explicitly list their test content in a pre-determined format either generated via an external process, embedded within the test adapter configuration or else fall-back to the list content discovery mechanism.
Any and all discovered tests by any of the mentioned approach are then identified to the Visual Studio APIs. Test suite information and label information is also provided to Visual Studio so that it is possible to categorize test cases via said traits.
Once tests are discovered from a module, the Boost Test Adapter attempts to execute test cases. Each test case is individually executed in a separate process so that test results can be communicated immediately back to Visual Studio unless specified otherwise via the <TestBatchStrategy> configuration value. Results generated by the process are later parsed and sent to Visual Studio. Note that from Visual Studio 2015 Update 1, tests may also be executed in parallel.
Test modules are executed as a separate process. On completion, test results (located within the user's temporary folder) are then parsed and communicated to Visual Studio. This process normally runs within the directory where the module is located i.e. the working directory is the same directory in which the executable is located and uses the general environment variables. These two properties can be modified via Visual Studio's Debugging property page or, in the case of working directory, via the .runsettings configuration.
Executing test cases individually can be slow, especially for code coverage since symbols need to be reloaded per process spawn. In an attempt to minimize this cost, tests are batched per test suite. Test cases contained within the same test suite are then executed in one go.