Skip to content
This repository has been archived by the owner on Jul 17, 2023. It is now read-only.

User Manual

Jump to bottom
Brian Gatt edited this page Nov 2, 2017 · 21 revisions

Boost Unit Test Adapter for Microsoft Visual Studio

User Manual

Contents

Introduction

The Boost Unit Test Adapter is available as a free extension for Microsoft Visual Studio. It makes use of the Unit Test Explorer (UTE) provided by Microsoft in the Visual Studio IDE to visualize and run unit test cases that are written using the open source Boost Unit Test Framework libraries. Boost provides free, peer-reviewed, portable C++ source libraries. Boost libraries are intended to be widely useful and usable across a broad spectrum of applications. Boost works on almost any modern operating system, including UNIX and Windows variants.

Refer to http://www.boost.org/ for detailed information about the Boost libraries.

Setup

Pre-Requisites

In order to use the Boost Unit Test Adapter, the following components must be available:

  • Boost Libraries. The following versions are supported:
    • Boost 1.60 and greater
  • Microsoft Visual Studio. The following versions are supported:
    • Visual Studio 2015 (Community, Professional, Enterprise).

Installation

Boost Unit Test Adapter Extension

The pre-built binary of the Boost Unit Test Adapter installation package ( .vsix file ) can be downloaded and installed either by:

  • downloading and installing the .vsix file via the Microsoft Visual Studio gallery.
  • searching for the tool via the Visual Studio's Extensions and Updates form via Tools -> Extensions and Updates and then clicking Download as shown in the below snippet. image
  • downloading and compiling the Boost Unit Adapter packages from this GitHub location.

Boost Unit Test Project Template

The Boost Unit Test Project template as well as the Boost Unit Test File template are installed as part of Boost Unit Test Adapter extension installation above.

Optional One-time Settings

The below mentioned settings are optional and are here being suggested as a good working practice and are by no way required by the Visual Studio Boost Unit Test adapter. These setting relate to the setup of the Boost include file paths and the Boost library paths. Rather then setting the paths inside the project's property pages, it is suggested to make use of environment variables.

In order to set these environment variables:

  1. Select advanced system settings in Control Panel
  2. Select advanced tab in system properties window
  3. Click environment variable button
  4. Click new button in User variable section (or at System Level)
  5. Give BOOST_INC as variable name and boost install directory as variable value. Eg: D:\boost_dir
  6. Create one more variable with BOOST_LIB as variable name and boost library directory as variable value. Eg: D:\boost_dir\lib

In case multiple Boost library versions are utilized, when naming the environment variables it is suggested to include the library version as part of the environment variable name. The same concept should also be applied with the processor architecture type the library is targeted for. Some examples are the following BOOST_1610_INC, BOOST_1610_LIB, BOOST_1610_INC_X64, BOOST_1610_LIB_X64

Uninstallation

  1. Open Microsoft Visual Studio.
  2. Go to Tools -> Extensions and Updates.
  3. Select Boost Unit Test Adapter.
  4. Click Uninstall.
  5. Restart Microsoft Visual Studio.

Quick start

  • Double click BoostTestPlugin.vsix. This will install the Boost Unit Test Project template, Boost Unit Test File template and Boost Unit Test Adapter extension.
  • Follow the optional installation steps as described in section Other optional settings so as to create the environment variables. This is a one time process so if this has already been done, this step can be skipped.
  • Create a new command line project which will contain your Boost.Test code.
  • For easier setup, use the Boost Unit Test Project template available in the Add New Project dialog. This can be found under: Installed -> Visual C++ -> Test -> Boost Unit Test Project.
    • This will result in a new solution having a project which contains 2 header files (viz. stdafx.h and targetver.h) and 3 source files (viz. stdafx.cpp, BoostUnitTestSample.cpp and BoostUnitTest.cpp).
    • Set the Platform Toolset property value of the newly added project. This can be found under: Project Property Pages -> Configuration Properties -> General -> Platform Toolset.
  • Build the solution and make sure that the project built correctly.
  • Once a successful build is achieved, the test cases should have been discovered and made visible in the Test Explorer Window. (If the Test Explorer Window is not visible, it can be accessed via the menu item Test -> Windows -> Test Explorer).
  • The test cases appear with a light blue icon, indicating that they are newly discovered.
  • Run the test cases by using the Run All option. As a result, the units tests cases are executed and the corresponding results are updated. The green tick symbolizes that the test case has passed. In case the test case failed (due to failed assertions or memory leaks) the test icon will be red. In case of other failures, the icon will be a yellow triangle.
  • Selecting a test case will provide additional information regarding execution duration, failed assertions, test output and test duration.

Usage

Add a New Boost Project Using the Boost Unit Test Project Template

A new project can be added using the Boost Unit Test Project template available in the Add New Project context menu. This can be found under Installed -> Visual C++ -> Test -> Boost Unit Test Project as shown in the below snippet.

image

This will result in the creation of a new solution having a project which contains 2 header files (viz. stdafx.h and targetver.h) and 3 source files (viz. stdafx.cpp, BoostUnitTestSample.cpp and BoostUnitTest.cpp) as shown below.

image

The relevant platform toolset property value need to set next for the newly created project. This property can be accessed via the Property Pages -> Configuration Properties -> General -> Platform Toolset as shown also in the below snippet.

image

It is important to note that by default the Boost Unit Test Project template does not have any value set for Platform Toolset property. This means that by default, the property value is set to whatever Visual Studio version is the project being created and compiled with. Failure to properly set the Platform Toolset version might cause linking failures in case any compiled libraries are not available for the Platform Toolset version selected.

As soon as a Boost Project is newly created using the Boost Unit Test Project template, it is suggested that the solution is built so as to make sure that no compilation or linking issues occur. The most common issue is that users might have incorrect references of either the Boost includes path or the Boost library paths, so in case of compiler warnings of include files not found, undefined identifiers or linker issues, it is suggested to check the Additional Include Directories and the Additional Library Directories as shown in the below two snippets. The configuration shown in the below two snippets assume that the steps as indicated in section Optional one time settings have been followed. In case not, the user has to write the paths of the Boost include directories and the Boost library directory respectively.

image

image

Another common pitfall is having an incorrectly set Platform Toolset version. This generally occurs because in case the Boost pre-compiled binaries are utilized, some Boost libraries might not be available for certain levels of Platform Toolset utilized. In such case it is suggested to change the Platform Toolset version according the Platform Toolset the library has been built for, which can be generally easily identified via the library filename(s) as shown in the below snippet.

image

Once a successful build is achieved, the test cases should have been discovered and should be visible in the Test Explorer Window. (If the Test Explorer Window is not visible, it can be accessed here via the menu item Test -> Windows -> Test Explorer as shown in the below snippet.)

image

Add a New Boost Unit Test File Using the Boost Unit Test File Template

A new file can be added using the Boost Unit Test File template available via the context menu Add -> New Item and selecting Boost Unit Test File under Visual C++->Test dialog as shown in the below two snippets.

image
image

Display Test Cases

In order for the Boost Unit Test Adapter to enumerate the list of available unit tests in a Boost Unit Test project, the project should be successfully built first. Once the project is successfully built, the test cases are enumerated in the Test Explorer window as shown in the below snippet.

image

If the Test Explorer Window is not visible, it can be accessed here via the menu item Test -> Windows -> Test Explorer as shown in the below snippet

image

Run All Test Cases

When the Run All button is clicked, all enabled test cases are executed and test explorer window is updated accordingly. The passed and failed test cases are listed separately. Successful unit tests are indicated via a green success icon while failed tests are indicated via a red failed icon. In case a discovered test case cannot executed or located, the test will be flagged with a yellow warning triangle.

image

The execution of the tests can be monitored by selecting Tests in the drop down menu available in the Output window.

Note that pressing Run All will not execute tests which are explicitly disabled using Boost.Test's utf::disabled() and utf::enabled_if_ decorators. For more details, see Force Run Explicitly Disabled Tests.

Run Selected Test Cases

Test cases can be executed individually by selecting the desired test cases, then right clicking on them and then clicking the Run Selected Tests option. Their execution can again be monitored by selecting Tests in the drop down menu available in the Output window.

Note that this functionality also works on disabled test cases, that is, they will still be run if they are explicitly selected.

image

Group Test Cases

The test cases can be grouped by any of the four options given in the table below, by selecting the Group By menu item in the Test Explorer Window and selecting the desired option.

Group Type Description
Project Groups test cases according to its project’s name.
Traits/Suites Groups test cases according to the suite and label it belongs to, as well as by whether they are enabled or disabled.
Outcome Groups test cases by execution results: Failed Tests, Skipped Tests, and Passed Tests.
Duration Groups test cases by execution time: Fast (<100 ms), Medium (>100 ms), and Slow (>1 sec).
Class Unsupported by the Boost Unit Test Adapter.

  • Grouping by Project:
    image

  • Grouping by Traits/Suites:
    By selecting the Traits option in the Test Explorer Window, the test cases are grouped via the test suite they fall under, any labels which are used as decoration as well as their status (enabled or disabled). If any of the test cases does not belong to any test suite, then it falls under the Master Test Suite. image

  • Grouping by Outcome:
    By default, the test cases are grouped by outcome, with execution times listed. image

  • Grouping by Duration:
    image

Find the Test Case Location in Source

The source file name as well as the line number of a unit test is displayed in the lower pane of the Test Explorer Window. Clicking on these hyperlinks will focus the source code window on the file and line in question. Additionally, double-clicking a test case will also navigate to the test case source code.

image

Search and Filter the Test Case List

When you type a string in the Test Explorer search box and press ENTER, the test case list is filtered to display only those test cases whose fully qualified names contain the string being searched. More advanced filtering can be done on also other aspects of the tests (such as on the Output) by using the search filters provided by Visual Studio.

image

Clear the Search Results

The X button available on the right of the search filter clears the search box's content and resets the Test Explorer's test filter.

image

View the Summary and Output of the Test Case Results

After a unit test has been executed, single-click the test case to view the summary of its results. The Output hyperlink will focus on a new window showing the output generated during test execution. Information contained in the Output window include:

  • Standard Output Messages
  • Standard Error Messages
  • Failed Assertions
  • Memory Leaks

image

Find the Error Location in Source

For tests which fail, apart from the Output hyperlink, a StackTrace hyperlink is also shown. Clicking on this hyperlink will focus the source code window on the file and line number of the last error which caused the test to fail.

image

Add and Run a Playlist

Test cases can be grouped using the Playlist feature. After selecting a series of test cases, use the right-click context menu to save the tests in a playlist as shown below.

image

By selecting the Playlist option in the Test Explorer window, all the test cases of the particular playlist are displayed and can be executed.

image

Debug Unit Tests Using Test Explorer Window

The unit test cases can be debugged using the Test Explorer Window in the following manner:

  1. Select the test cases to be debugged. Assuming breakpoints have been already set in the code that is going to be debugged, right click on the applicable test case and select Debug Selected Tests.

image

  1. The debug point would get hit and then the unit tests can be debugged normally via the Visual Studio debug functions/keyboard shortcuts.
  2. After the debug run is complete, the result of the test case is shown in the Test Explorer Window.
  3. If all the test cases in the solution need to be debugged, go to: Test -> Debug -> All Tests. All the test cases can be debugged and the result would be displayed in the Test Explorer Window.

Detect from within a unit test that the test is being executed using the Boost Unit Test Adapter

Whenever a Boost Unit Test is executed using the Boost Unit Test Adapter, the environment variable BUTA is set. A developer might find this useful, in case he/she wants to increase the logging verbosity generated to the standard output by his/her code (not to be confused with the one generated by the Boost unit test framework) only when the Boost Unit Test Adapter is executing the test.

Analyzing Code Coverage for Boost Tests

[Available for MS Visual Studio 2015 Enterprise]
Code coverage can be analyzed via the Boost unit tests discovered through the test adapter. By navigating to the Test -> Analyze Code Coverage -> All Tests menu option from the Visual Studio menu bar, all discovered unit tests are executed and the debugger will analyze the amount of code covered by the executed tests.

image

Apart from analyzing code coverage of all the tests, tests can be individually selected and code coverage metrics can be identified for the selected subset via the right-click context menu or via the Test -> Analyze Code Coverage -> Selected Tests menu option.

Note that code coverage does not report standard output, standard error and memory leak information for executed tests. In order to minimise executable re-loading, tests are executed in batches and as a result, standard output, standard error and memory leaks cannot be distinguished per test case. The difference in the execution behaviour can be seen via the Tests output log window as shown in the snippets below.

image

image

The top snippet shows the tests being executed not for code coverage analysis purposes and as it can be noticed the tests are executed individually. This is done so as to permit the extraction of the standard output and the detection of memory leaks.

The bottom snippet shows the tests being executed for code coverage purposes and as it can be noticed the tests are executed in a batch, grouped using the most suitable strategy.

For further information about code coverage in Visual Studio, please refer to the Using Code Coverage to Determine How Much Code is being Tested MSDN article.

Boost Unit Test Adapter Configuration

The Boost Unit Test adapter can be configured via a Visual Studio .runsettings configuration file. Selecting and using an appropriate test settings file containing a valid <BoostTest> section allows for running tests which cater for specific use-cases. The configuration file can be loaded via the Visual Studio menu option Test -> Test Settings -> Select Test Settings File as shown in the below snippet. In general, omitting a configuration option implies that its default value is taken into consideration.

image

A sample settings file is available in the repository. The supported configuration options will be described in the following sub sections.

Failing Tests in case Memory Leaks are Detected

If tests are required to fail in case memory leaks are detected, the configuration option <FailTestOnMemoryLeak> should be set to true (or 1). By default, this configuration option is set to false (or 0).

As soon as tests are executed, in case memory leaks are detected, the specific test with leaking memory is reported as failed as shown in the below snippet. If the user clicks on the Output link, the user can see the memory allocation number detail, the leak size and the leak contents.

image

Specifying a Test Execution Timeout

Limiting test execution time can be configured via the <ExecutionTimeoutMilliseconds> configuration option. The configuration value specifies the amount of time in milliseconds each test case is allowed to run. By default, this option is set to -1 implying that tests are allowed to run indefinitely.

The below snippet shows a test scenario where a test execution lasted more than the configured 1000ms (i.e. 1 second)

image

Modifying Boost Test Log Verbosity

To modify the amount of log information displayed in the test adapter, the <LogLevel> configuration option needs to be explicitly set to one of the following values:

  • All
  • Success
  • TestSuite
  • Message
  • Warning
  • Error
  • CppException
  • SystemError
  • FatalError
  • Nothing

By default, this option is set to TestSuite. Please refer to the log_level Boost Runtime Configuration for further information about log verbosity.

Disable Catching of System Errors

By default, the Boost UTF and the Boost unit test adapter catches system errors. To disable this behaviour, set the <CatchSystemErrors> configuration option to false (or 0).

Please refer to the catch_system_errors Boost Runtime Configuration for further information.

Enable Floating Point Exceptions

The Boost UTF framework allows for trapping floating point exceptions. In order to enable this functionality through the Boost unit test adapter, set the <DetectFloatingPointExceptions> configuration option to true (or 1).

Please refer to the detect_fp_exceptions Boost Runtime Configuration for further information.

Configure Test Batching Strategy

By default, the Boost Unit Test Adapter will execute each and every test case in an individual process. If this is behaviour is not to one's liking, this can be modified via the <TestBatchStrategy> configuration option. This can be set to either:

Test Batching Strategy Description
Source All tests defined within the same test module (executable/dll) are executed in one process.
One All selected test cases are executed in one process should the test module be a valid Boost 1.63 module (otherwise falls-back to TestSuite strategy).
TestSuite All tests defined within the same test suite are executed in one process.
TestCase Default behaviour. Test cases are executed in an individual process.

Filtering Tests

The Boost Test Adapter is notified once an exe or dll module is successfully compiled by Visual Studio in order to discover tests. To avoid searching all possible compiled modules (which is a common scenario when using solutions consisting of both tests and non-test modules), modules can be filtered via <Filters>. Filters are regular-expression patterns applied on a compiled modules fully-qualified path (e.g. C:\path\to\test.exe) and can be used to determine whether a module is included or excluded.

For further information on how to configure filters, please refer to sample .runsettings configuration.

Force Run Explicitly Disabled Tests

By default, tests which are explicitly disabled using the Boost decorators utf::disabled() and uft::enable_if are not executed when Run All is pressed. By setting the <RunDisabledTests> option to true in the .runsettings file, any explicitly disabled tests will also be run when Run All is pressed.

Force List Content Discovery

In order to deem a test module executable as safe, the Boost Unit Test Adapter will check the generated executable for Boost.Test eye-catchers. Unfortunately, this does not work if Boost.Test is dynamically linked. In order to force the use of --list_content=DOT and benefit from improved discovery, set the <ForceBoostVersion> configuration option to the Boost version which was used for the development of the test module.

Alternatively, should the test module file name (including extension) ends in test.boostd.exe or test.boost.exe, the adapter assumes that the module is a safe Boost.Test module. Note that the <ForceBoostVersion> configuration element is implied to be set as <ForceBoostVersion>1.59</ForceBoostVersion> if the test module ends in either test.boostd.exe or test.boost.exe.

Utilization of an External Test Runner

The Boost Unit Test Adapter supports the utilization of an External Test Runner for test discovery and test execution. This is useful in case a user, for example, wants to compile the Boost unit tests project as a dll rather than as an executable or should the test module require additional command-line arguments for correct operation.

The example configuration described in this section will make use of the sister project Boost External Test Runner available at GitHub but a user can use any other compatible Boost Test Runner as long as the test results are in the same format as those generated typically by the Boost Unit Test Framework. In addition, with respect to the test execution aspect, the same argument structure as the Boost Unit Test Framework needs to be supported.

Tests Discovery Configuration

If an external test runner is used, the discovery process will still utilise the --list_content=DOT feature, however, all the work will be handled by the external test runner. For further details, please refer to the sample configuration.

The command generated is similar to the following, where the sub-string {source} is replaced with the full path of the test module and {boost-args} with the Boost.Test arguments required for --list_content=DOT. For example, given the following command-line configuration: C:\PROGRA~2\BoostExternalTestRunner.exe --test "{source}" {boost-args}, the following command-line will be eventually executed for discovery purposes:

C:\PROGRA~2\BoostExternalTestRunner.exe –-test "D:\dev\svn\SampleBoostProject\Debug\TestProject.dll" --init "init_unit_test" "--list_content=DOT" > "SpecialCharacters.exe.test.stdout.log" 2> "SpecialCharacters.exe.test.stderr.log"
Tests Execution Configuration

The configuration of the execution phase is performed via the information element <ExecutionCommandLine> as shown in the .runsettings XML configuration snippets shown below.

<ExternalTestRunner type=".dll">
    <ExecutionCommandLine>C:\PROGRA~2\BoostExternalTestRunner.exe --test "{source}" {boost-args}</ExecutionCommandLine>
 </ExternalTestRunner>

The Boost Unit Test Adapter will replace the sub-string {source} with the full path and name of the project and {boost-args} with the boost arguments so as to run the test. An example of the command generated is

C:\PROGRA~2\BoostExternalTestRunner.exe –-test "D:\dev\svn\SampleBoostProject\Debug\TestProject.dll" "--run_test=SpecialCharactersInIdentifier" "--log_format=xml" "--log_level=test_suite" "--log_sink=SpecialCharacters.exe.test.log.xml" "--report_format=xml" "--report_level=detailed" "--report_sink=SpecialCharacters.exe.test.report.xml" > "SpecialCharacters.exe.test.stdout.log" 2> "SpecialCharacters.exe.test.stderr.log"

The Boost Unit Test Adapter will issue the generated command and, upon process completion, it will read the generated tests results in XML format (in format that is typically generated by Boost Unit Test Framework XML format) along with the standard output and standard error stream output and then display the test results accordingly in the Test Explorer window.

Limitations

  • If the log redirection functionality of Boost.Test is used, the Boost Unit Test Adapter will not work. This is due to the fact that adapter internally uses the redirected log output.

License

Boost Unit Test Adapter License

The Boost Unit Test Adapter is released under the Boost Software Licence - Version 1.0 - August 17th, 2003.

Third-party Software Credits

The Boost Unit Test Adapter makes use of the below list of dynamically linked libraries for which we would like to thank their contributors:

Appendix

Version history

The version history is maintained at the following link.

Reporting Issues

Reporting Issues can be done at the following link

Troubleshooting

The troubleshooting page is maintained at the following link.

Home

User Manual
Configuration
Version History
Troubleshooting
FAQ

Boost Unit Test Adapter Design
Building From Sources
Git Workflow
Testing

Continuous Integration

Clone this wiki locally