Measuring Code Coverage on Firefox

What is Code Coverage?

Code coverage essentially is about measuring how often certain lines are hit, branches taken or conditions met in a program, given some test that you run on it. There are different types of coverage metrics (see also the Wikipedia entry), but when we speak of code coverage here, we usually mean line and branch coverage. This type of coverage is only concerned with hit counts for lines and branches.

What Code Coverage tells us, and what it doesn't

The question is not easy to answer comprehensively, but there are two very important things that code coverage can, and cannot tell us:

• If a certain branch of code is not hit at all while running your tests, then you will never be able to find a bug in this particular piece of the code using these tests.
• If a certain branch of code is executed (even very often), this still doesn't tell you about the quality of your test. It could well be that a test exercises the code but does not actually check that the code performs correctly.

As a conclusion, we can use code coverage to find areas that need (more) tests, but we cannot use it to confirm that certain areas are well tested.

C/C++ Code Coverage on Firefox

There are several ways to get C/C++ coverage information for mozilla-central, including creating your own coverage builds. The next sections describe the available options.

Generate Code Coverage report from a try build (or any other treeherder build)

To spin a code coverage build, you need to select the linux64-ccov platform. E.g. for a try build:

./mach try -b o -p linux64-ccov -u all -t none

There are two options now, you can either generate the report locally or use a one-click loaner.

Generate report using a one-click loaner

Select the B job on Treeherder and get a one-click loaner.

In the loaner, download and execute the script https://github.com/marco-c/firefox-code-coverage/blob/master/codecoverage.py:

wget https://raw.githubusercontent.com/marco-c/firefox-code-coverage/master/codecoverage.py
python2.7 codecoverage.py

This command will automatically generate a HTML report of the code coverage information in the report subdirectory in your current working directory.

Generate report locally

Prerequisites:

• Download the Python script at https://github.com/marco-c/firefox-code-coverage/blob/master/codecoverage.py.

Given a treeherder linux64-ccov build (with its branch, e.g. `mozilla-central` or `try`, and revision), run the following command:

python codecoverage.py PATH/TO/MOZILLA/SRC/DIR/ BRANCH REVISION

This command will automatically download code coverage artifacts from the treeherder build and generate an HTML report of the code coverage information. The report will be stored in the report subdirectory in your current working directory.

Creating your own Coverage Build

On Linux and Mac OS X it is straightforward to generate a gcov build using GCC. Adding the following lines to your .mozconfig file should be sufficient:

# Enable code coverage
ac_add_options --enable-coverage
export CFLAGS="--coverage"
export CXXFLAGS="--coverage"
export LDFLAGS="--coverage -lgcov"
# Needed for e10s:
# Without debug mode, content processes terminate with "_exit", which doesn't update coverage counters.
ac_add_options --enable-debug
# With the sandbox, content processes can't write updated coverage counters in the gcda files.
ac_add_options --disable-sandbox

The --coverage flag replaces two older flags, -fprofile-arcs -ftest-coverage.  Also, make sure you are not running with artifact builds enabled, as it can prevent coverage artifacts from being created.

It is recommended to generate code coverage data without optimizations (see Using gcov with GCC Optimization), by adding the following to your mozconfig:

ac_add_options --disable-optimize

You can then create your build as usual. Once the build is complete, you can run any tests/tools you would like to run and the coverage data gets automatically written to special files. In order to view/process this data, we recommend using the lcov tool, a tool to manage and visualize gcov results:

export COVERAGE_DIR=../coverage
# Create directory to hold your codecoverage results
mkdir $COVERAGE_DIR
# Collect all the coverage data from the directory YOUR_OBJ_DIR/PATH_OF_INTEREST; you can leave PATH_OF_INTEREST blank if you want code coverage information for the entire codebase.
lcov -b . --directory YOUR_OBJ_DIR/PATH_OF_INTEREST --capture --ignore-errors source,graph --output-file $COVERAGE_DIR/coverage.info
# This generates a HTML report from the coverage.info file.
genhtml -o $COVERAGE_DIR --show-details --highlight --ignore-errors source --legend $COVERAGE_DIR/coverage.info

Once you have created HTML output from the coverage data, you can easily view it by pointing your browser to the output directory. The lcov tool also allows you to reset all coverage data to do multiple runs, combine several coverage files and various other tasks. For more information see man lcov and man genhtml.

Debugging Failing Tests on the Try Server

When code coverage is run through a push to try, all the data that is created is ingested by ActiveData and processed into a different data format for analysis. Anytime a code coverage run generates *.gcda and *.gcno files, ActiveData starts working. Now sometimes, a test will permanently fail when it is running on a build that is instrumented with GCOV. To debug these issues without overloading ActiveData with garbage coverage data, there are two things that need to be done to prevent artifacts from being uploaded. First, open the linux64-ccov code-coverage mozconfig. In there, remove the following:

ac_add_options --enable-coverage

Next, open the file taskcluster/taskgraph/transforms/tests.py and add the following line,

test['mozharness'].setdefault('extra-options', []).append('--disable-ccov-upload')

right after this line of code:

test['mozharness'].setdefault('extra-options', []).append('--code-coverage')

Now when you push to try to debug some failing tests, or anything else, there will not be any code coverage artifacts uploaded from the build machines or from the test machines.

JS Debugger Per Test Code Coverage on Firefox

There are two ways to get javascript per test code coverage information for mozilla-central. The next sections describe these options.

Generate Per Test Code Coverage from a try build (or any other treeherder build)

To spin a code coverage build, you need to select the linux64-jsdcov platform. E.g. for a try build:

./mach try -b o -p linux64-jsdcov -u all -t none

This produces JavaScript Object Notation (JSON) files that can be downloaded from the treeherder testing machines and processed or analyzed locally.

Generate Per Test Code Coverage Locally

To generate the JSON files containg coverage information locally, simply add an extra argument called --jscov-dir-prefix which accepts a directory as it's input and stores the resulting data in that directory. For example, to collect code coverage for the entire Mochitest suite:

./mach mochitest --jscov-dir-prefix /PATH/TO/COVERAGE/DIR/

Currently, only the Mochitest and Xpcshell test suites have this capability.