Skip to content
This repository has been archived by the owner on Dec 29, 2021. It is now read-only.

Coverage reporting missing for binaries under-test #104

Closed
epage opened this issue May 8, 2018 · 4 comments
Closed

Coverage reporting missing for binaries under-test #104

epage opened this issue May 8, 2018 · 4 comments

Comments

@epage
Copy link
Collaborator

epage commented May 8, 2018

@mssun

Another issue about testing CLIs is to correctly calculate code coverage. If using std::process::Command to fork and execute a CLI, all existing code coverage tools (tarpaulin, gcov, kcov, etc) will failed to calculate the statistics. There is not a better way to properly calculate the line coverage including unit tests and integration tests.

From https://github.com/rust-lang-nursery/cli-wg/issues/9#issuecomment-385487756

@epage
Copy link
Collaborator Author

epage commented May 8, 2018

@mssun That is a good point but I'm expecting that won't be too much of an issue. End-to-end testing using assert_cli should, ideally, be reserved for the parts of the application that cannot be tested at a lower level. The coverage will be lower and a developer will need to be aware of what is or isn't being tested because of these end-to-end tests.

If there is an enterprising person who has an idea on how to solve that, great! Otherwise, we should call this out as an known issue in assert_clis documentation.

@epage
Copy link
Collaborator Author

epage commented May 8, 2018

@mssun

Ideally, we can use gcov to solve this issue like this https://jbp.io/2017/07/19/measuring-test-coverage-of-rust-programs.html. However, it is not as easy as I thought.

It is impossible to build both binary and tests with -Copt-level=1 -Clink-dead-code -Ccodegen-units=1 -Zno-landing-pads -Cpasses=insert-gcov-profiling -L/Library/Developer/CommandLineTools/usr/lib/clang/9.1.0/lib/darwin/ -lclang_rt.profile_osx flags only.

Because this command cargo rustc --test tests -vv -- -Copt-level=1 -Clink-dead-code -Ccodegen-units=1 -Zno-landing-pads -Cpasses=insert-gcov-profiling -L/Library/Developer/CommandLineTools/usr/lib/clang/9.1.0/lib/darwin/ -lclang_rt.profile_osx will only build test with the flags.

And using the RUSTFLAGS environment variable won't solve this problem. It will add compilation flags to all crates. This will generate gcna and gcno files for other non-related crates in sysroot. This may also introduce some bugs of gcov when merging gcno (still don't know why).

Overall, I think the testing coverage issue is related to many parties such as: cargo, rustc, coverage tools, etc.

@mssun
Copy link

mssun commented May 9, 2018

Cool, thanks.

@epage
Copy link
Collaborator Author

epage commented May 29, 2018

The responsibility for this is moving to assert_cmd and tis will be addressed in assert-rs/assert_cmd#9

assert_cli is going to morph into a wider-focused tool, including things like assert_fs. See #41

@epage epage closed this as completed May 29, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants