Debugging B2G using Valgrind

ValgrindDSS's Memcheck tool detects C++ memory management errors that can lead to stability and security problems: use of freed memory, heap array overruns, uses of uninitialised values, and memory leaks. It is included in the Firefox OS tree, and can be run on certain phones that have adequate resources. This article explains how to use it.

Requirements

Before running Valgrind against Firefox OS, it's recommended that developers familiarize themselves with the Debugging Mozilla with ValgrindDSDDSDDSG article. Most of the information it contains is relevant to running against Firefox OS, though many of the build and command line steps are taken care of by the Firefox OS build and execution scripts.

To run Valgrind against Firefox OS on a phone, a phone with higher than normal specs is required. Valgrind is resource intensive, and will generally cause Out Of Memory errors on phones with anything less than 1GB of RAM. As of this writing (2015-06-10), Valgrind is known to work with Firefox OS running on Flame phones with 1GB or RAM and Nexus 5 phones with 2GB of RAM. It should also run on Geeksphone Keons and similar spec phones. Look at Phone and device data for more details of the phones available.

The number one problem is running out of memory. If your phone's kernel supports swap space, your situation is improved. You can tell whether your phone supports swap space by looking for the file /proc/swaps -- if it is present, you are in luck. If so, prepare a swap file on the host, move it to the phone, and enable it:

dd if=/dev/zero of=swapfile800M bs=1024 count=781250
mkswap swapfile800M
gzip --fast swapfile800M
adb push swapfile800M.gz /data
adb shell gzip -d /data/swapfile800M.gz
adb shell swapon /data/swapfile800M
adb shell cat /proc/swaps
# should produce a line like this:
# /data/swapfile800M                      file        781244    0    -2

If your phone doesn't have swap, you may experience Nuwa-created processes being OOM-killed, and hence incomplete or confusing functionality. Try to use a phone with at least 2GB of RAM in this case. Flame, at least, does support swap. You need to be careful which partition the swap file is placed on. If you inadvertantly put it on a btrfs partition, you'll get complaints about holes in swap files when you try to run swapon.

Running Valgrind on FxOS Phones

Let's look at the process of running Valgrind.

Compiling

To build Firefox OS with valgrind enabled, add the following to to the .userconfig file.

export B2G_VALGRIND=1
export DISABLE_JEMALLOC=1

Building debug (B2G_DEBUG) is also recommended. Building with no optimizations (B2G_NOOPT) makes things run almost unusably slow, and is not recommended except in cases where it is thought that optimizations may be obfuscating errors.

And add the following to the end of gonk-misc/default-gecko-config:

ac_add_options --enable-optimize="-g -O2"
ac_add_options --enable-valgrind
ac_add_options --disable-jemalloc
ac_add_options --disable-sandbox

Disabling sandboxing is unfortunately required, since not doing so causes the Valgrind processes to be killed by the sandbox mechanism.

Running

Note: Running Valgrind on a Firefox OS phone is done in the context of the phone, not the host operating system. This means that developers can use any platform that has adb available and will execute the run-valgrind.sh script to run valgrind on the phone.

To run Firefox OS under valgrind, use the run-valgrind.sh script from the B2G directory. This script does the following:

Remounts the phone file system as r/w.
Copies the current debug libxul.so with full symbols to the phone. As this file is many hundreds of MB, this step can take about two minutes to finish. It needs to be redone every time a new build is made. To run valgrind without the libxul copy step, run this command:
```
run-valgrind.sh nocopy
```
Reboots the phone.
Kills the b2g process that starts up on phone startup.
Runs its own b2g process under valgrind.

All valgrind output will be written to the stdout of the terminal executing the run-valgrind.sh script. This can either be read in the terminal or piped to a file.

run-valgrind.sh starts Valgrind with a suitable set of command line parameters. If you want to pass in extra parameters that override the default set, specify them using the EXTRA_ARGS environment variable:

EXTRA_ARGS="-v" run-valgrind.sh nocopy

Note: Since the run-valgrind.sh script owns the adb process running the b2g process, killing the script will also kill b2g and valgrind on the phone. It is recommended that the phone be rebooted after running a valgrind session, as it can leave things in an odd state.

Debug info for system libraries

To get good quality stack traces for system libraries, you need to put them on the phone:

(cd out/target/product/flame && adb push symbols /sdcard/symbols-for-valgrind)

run-valgrind.sh will automatically cause Valgrind to read the debug objects in /sdcard/symbols-for-valgrind. If you want to verify that Valgrind is reading them, start run-valgrind.sh with EXTRA_ARGS="-v".

Running Valgrind on Firefox OS Desktop

Running valgrind against Firefox OS Desktop works the same way as running it against desktop Firefox. Consult the Debugging Mozilla with ValgrindDDS page for more information. All of the relevant build flags will need to be added to mozconfig, and all platform-specific issues in the page will apply.

Note that running Valgrind on the desktop in OOP/process-per-tab mode will require adding the following option to make sure child processes are also traced:

--trace-children=yes

Maintaining and Updating Firefox OS Valgrind

While patches are upstreamed when applicable, Valgrind for Firefox OS is maintained in a forked repository to keep things as up to date as possible while also dealing with the eccentricities of the Firefox OS build tree and versions.

Updating Valgrind Repos

WARNING: EXPERIENCE WITH GIT REQUIRED. Do not attempt to upgrade the valgrind repos if you are not familiar with handling complex Git operations. Any updates to the github repo will mirror to git.mozilla.org, which will then be pulled by developers using the HEAD of the manifests repo. While breaking Valgrind will not break builds on anything that does not have Valgrind enabled (e.g. Buildbot automation), it will very much annoy developers who are trying to use it.

The main Firefox OS valgrind and VEX repos are at

The master branch is a pristine version of the SVN trunk of each of these repos, while the Firefox OS branch contains Firefox OS-specific patches rebased on top of the trunk.

Note: ALWAYS UPDATE BOTH REPOS AT THE SAME TIME. While they are two seperate repos, VEX is usually a submodule of Valgrind, and the HEAD of valgrind usually points to the HEAD of VEX.

These are replicated onto the git.mozilla.org domain for use in B2G manifests:

The main repos are kept in sync with the valgrind SVN with git svn. To pull updates to the repos, clone the valgrind and vex repos from github, then run the following:
```
git svn init -s [subversion repo url]
```
Pulling the SVN info will take hours, but when it is done, your tree should be in sync with the main Valgrind SVN.

To pull further updates, the following set of commands is used:

git checkout master
git svn fetch
git svn rebase
git push [github-remote-name] master
git checkout fxos
git rebase master

There is a good chance that there will be patch conflicts during the Firefox OS branch rebase step. If you cannot work the issue out, email the author of the conflicting commit.
After rebasing, run a full Firefox OS build with the B2G_VALGRIND flag to make sure that it still builds. The most common required fixes are listed in the scripts section below.
Once you have rebased AND TESTED the build against the Firefox OS tree, you will have to force push the Firefox OS branch due to head change.
```
git push -f [github-remote-name] fxos
```

Build, Install, and Execution Scripts

There are multiple scripts that are either part of the Valgrind Firefox OS branch or B2G repo that may need to be updated after a repo fetch.

external/valgrind/android.mk

This is the Android build system script. More often than not, this is where changes will need to be made, due to files being added/removed from the Valgrind tree. Use a -j1 build to see which target is failing to build, and if it is missing a file or referencing a non-existent file, update that project's file list.

external/valgrind/valgrind.mk

This contains the list of packages that need to be built and added to the FxOS system image, referenced by gonk-misc/b2g.mk. This usually needs no update as it is rare for Valgrind to add new packages, but if they're needed, put them here.

run-valgrind.sh

The script for running Valgrind on the phone. If there are new command line arguments that are needed to run Valgrind on the phone, put them here. This is also where we copy the library with debug symbols to the phone, so any adjustment/change to that process should happen here.