How to properly use coverage.py in Python?

Question

I've just started using Coverage.py module and so decided to make a simple test to check how it works.

Sample.py

def sum(num1, num2):
    return num1 + num2


def sum_only_positive(num1, num2):
    if num1 > 0 and num2 > 0:
        return num1 + num2
    else:
        return None

test.py

from sample import sum, sum_only_positive

def test_sum():
    assert sum(5, 5) == 10

def test_sum_positive_ok():
    assert sum_only_positive(2, 2) == 4

def test_sum_positive_fail():
    assert sum_only_positive(-1, 2) is None

As you see, all my code is covered with tests and py.test says all of them pass. I expect Coverage.py to show 100% coverage. Well, no.

Well, Coverage.py may not see test.py file, so I copied test functions to sample.py file and ran Coverage again:

Then I added this block of code:

if __name__ == "__main__":
    print(sum(2, 4))
    print(sum_only_positive(2, 4))
    print(sum_only_positive(-1, 3))

and removed all test functions. After that, Coverage.py shows 100%:

Why is it so? Shouldn't Coverage.py show code test coverage, not just execution coverage? I've read an official F.A.Q. for Coverage.py, but can't find the solution.
Since many SO users are familiar with code testing and code coverage, I hope you can tell me, where am I mistaken.

I have just one thought here: Coverage.py may simply watch which lines of code aren't executed so I should write tests for those lines. But there're lines which are executed already but aren't covered with tests so Coverage.py will fail here.

Answer 1

Coverage looks for a .coverage file to read and generate that report for you. Py.test on its own does not create one. You need py.test plugin for coverage:

pip install pytest-cov

If you already have it, then you can run both at once like this:

py.test test.py --cov=sample.py

Which means run test module test.py and record/display coverage report on sample.py.

If you need to have multiple test runs and accumulate their recorded coverage and then display a final report, you can run it like this:

py.test test.py --cov=sample.py --cov-report=
py.test test.py --cov=sample2.py --cov-report=
py.test test.py --cov=sample3.py --cov-report=

Which means run test module test.py and record (only) coverage on sample.py - don't display a report.

Now you can run coverage command separately for a complete report:

coverage report -m

The command above simply displays a formatted coverage report based on the accumulated .coverage data file from previous test runs. -m means show lines missed i.e. lines not covered by tests:

Name        Stmts   Miss  Cover   Missing
-----------------------------------------
sample.py       6      0   100%  

Coverage supports more switches like --include and --omit to include/exclude files using path patterns. For more info check out their docs: https://coverage.readthedocs.io/en/coverage-4.5.1/cmd.html#reporting

Answer 2

It's a little hard to parse through your experiments, and you haven't included the command lines you used with each experiment. But: if you run the tests with:

python -m py.test test.py

then you can run them under coverage.py with:

coverage run -m py.test test.py