Infrastructure improvement: Add Coverage reporting

[wilfredinni]

Meaby is a good idea to add test coverage to the project. I did it in a fork and right now there is just a 42% covered.

[cool-RR]

Great work. @bittner How can we add this to our CI setup? (And hopefully with the HTML report that the coverage module generates.)

(I'd want to exclude third_party and include the tests package.)

[cool-RR]

Ah shit, looks like the coverage results are misleading, and fixing them wouldn't be trivial. This is because the coverage module uses sys.settrace to install a trace function, as does PySnooper. Therefore any lines that are run under PySnooper's tracer are excluded from the coverage report.

Perhaps it's possible to get our tracer to also run the coverage tracer?

@nedbat Is there a way to make it work?

[nedbat]

In theory it isn't hard: when you set your trace function, first use sys.gettrace to get the current trace function. In your function, call that one. There will still be gaps: the lines literally in your trace function will not be traced. I have the same problem trying to coverage-measure coverage.py.

[cool-RR]

@nedbat I tried that, and indeed had the problem of lines in my trace functions not getting traced. Of course, that's where all my critical lines are that I want to have coverage for.

I'm thinking about writing a PEP for this. I know this has been a long-time problem for people who write development tools, and I've asked my friends at Wingware how they're dealing with it. Imagine trying to develop a very complex debugger, without using a debugger!

I'm thinking that perhaps Python needs unlimited levels of tracers. Which means that instead of setting one tracer in settrace, there will be a list of tracers that you append to. Then, if I run PySnooper under coverage, we'll have two tracers. Once the PySnooper tracer kicks into action, only the coverage tracer keeps tracing.

If this works, then people could debug debuggers, measure coverage on debuggers, debug coverage tools, measure coverage on coverage tools, etc.

What do you think about that?

[nedbat]

I don't know what monsters await in trying to implement something like that. A different approach that I have considered is to unit-test my tracing code by calling it myself. That is, I have an idea how my trace() function should behave. I should be able to write tests that emulate Python calling it, and see if it does what I want.

The difficulty there is accurately emulating Python, especially in your case where you are examining data in the stack. One possibility there is to add another layer of complexity: a new trace function (trace_capture) would record the events and the state of the world it sees, then those events would be replayed to my trace function later, and I can coverage-measure that second execution. Again, there are lots of ways this could go wrong, so I have never experimented with it, and it's more complicated for your project than it is for mine.

Proposing that Python change, and then waiting for it to get implemented, seems like a very long path to what you need.

Perhaps the simplest thing you can do to increase the coverage measurement of PySnooper is to refactor your code so that as much of it as possible can be tested outside of the trace function.

[cool-RR]

Thanks Ned!

[bittner]

Great work. @bittner How can we add this to our CI setup? (And hopefully with the HTML report that the coverage module generates.)

Simply

• add a [testenv:coverage] section to tox.ini with the appropriate commands,
• add coverage to the envlist in the [tox] section and
• include it in the list of jobs in .travis.yml.

Without a dependency on an external service (e.g. Coveralls) this will be a bit less nice to consume, but still sufficiently useful.

[cool-RR]

Thanks. I think that until we can get a sensible result for coverage (i.e. inside the tracer), there isn't much point in doing that. If anyone's interested in working on it, go ahead.

[codexetreme]

Hey guys, I have just discovered this project and I gotta say I love it :) ... Can I help out by adding some tests and/or setting up CI/CD?? I will need some guidance regarding what are the end goals of this issue

[cool-RR]

This issue specifically is very difficult because of the difficulty of tracing tracers. Unless you're willing to jump into CPython's core and modify it to support multi level tracing. I suggest you take a look at the other issues, there's much more accessible work there. Feel free to pick one. Or more :)

…

On Thu, May 2, 2019, 20:38 Yashodhan Ghadge ***@***.***> wrote: Hey guys, I have just discovered this project and I gotta say I love it :) ... Can I help out by adding some tests and/or setting up CI/CD?? I will need some guidance regarding what are the end goals of this issue — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#46 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAN3SXSYT27YTLHPSJVMCDPTMRHDANCNFSM4HIIY5EA> .