Gaia unit tests

Running unit tests

You can run the unit tests on B2G desktop, Firefox Nightly or Mulet. You also need the latest Gaia repo. For most of the functionality, you must also have Node.js and NPM installed.

The Gaia repo contains a rather handy bin/gaia-test script, which helps running the tests in an easier way.

Launching the test runner in Mulet or Firefox

This will run the test server and launch your default Firefox as found in the path:

bin/gaia-test

Choosing Firefox binary

You can export the FIREFOX environment variable to your firefox binary. For example, on OS X:

export FIREFOX=/Applications/FirefoxNightly.app/Contents/MacOS/firefox

Alternatively, you can pass it as argument to bin/gaia-test:

bin/gaia-test <gaia directory> <firefox path>

Launching the test runner in B2G Desktop

This will download and launch B2G Desktop:

bin/gaia-test -d

Choosing B2G Desktop binary

You can use the environment variable B2G to point to your B2G Desktop binary:

export B2G=/home/user/gecko/obj-x86_64-unknown-linux-gnu/dist/bin/b2g

Alternatively, you can pass it as argument to bin/gaia-test:

bin/gaia-test <gaia directory> <B2G path>

Launching the test runner with gdb

Use the -g option:

bin/gaia-test -g

This works with both B2G Desktop and Firefox/Mulet. Of course it's a good idea to point to a binary built in debug mode (see above).

You can also use the environment variable GDB_ARGS to give more arguments to gdb:

GDB_ARGS="--symbols=<SYMFILE>" FIREFOX="<path to Firefox>" bin/gaia-test -g

After the usual prerequisite steps you'll end up in the gdb command line interface:

Reading symbols from /home/julien/firefox-nightly/firefox...(no debugging symbols found)...done.
(gdb)

Because several commands are launched in background you may have some other lines interleaved. You can just type ENTER to see the gdb command line on its own line.

Firefox uses the signal SIG38 and gdb stops at this signal, so you'll need to ask gdb to not stop before running the binary. You can enter this command to do just this:

(gdb) handle SIG38 nostop

Then add the breakpoints you want, and you can run the binary:

(gdb) r

Run the tests from the Web Interface

You can simply click on specific tests and then the Execute button.

Run the tests from the command line

With the WebSocket server running, and the Test Agent app running in B2G Desktop/Firefox Nightly, run the following command:

make test-agent-test

If you only want to run one app's tests you can specify which via the APP env variable:

make test-agent-test APP=calendar
# or
APP=calendar make test-agent-test

You can also optionally provide a reporter to format the test output:

REPORTER=List make test-agent-test

Run the tests as you save

When the server is running, the tests for a file are run automatically when a file is saved or even just touched:

• When you save a test file, the test file itself is run
• When you save another file, it finds a matching file in the test/unit directory, by suffixing the file name with _test.js.

Running tests like Buildbot does

Gaia unit tests in Buildbot are run using a separate runner; this explains how to use it.  Please consult the virtualenv docs if you're not familiar with using a Python virtualenv.

virtualenv venv
source venv/bin/activate
cd $GAIA/tests/python/gaia-unit-tests
python setup.py develop
cd gaia_unit_test
python main.py --binary /path/to/b2g/desktop/build --profile /path/to/gaia/profile

The Gaia profile must be made using the following:

NO_LOCK_SCREEN=1 DEBUG=1 DESKTOP=0 make

By default, this profile will be generated in $GAIA/profile-debug. bin/gaia-test generates the same profile so you don't need to regenerate it if you already run gaia-test.

Disabling a gaia unit test in automation

Automation uses a blacklist to exclude certain gaia unit tests from being run.  To prevent a test from running in automation, add its path to https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-unit-tests/gaia_unit_test/disabled.json.

Setting up your Gaia app

Although this guide should help make things easier, the best way to learn how to write, set up, and run tests is currently still to look at the source code; in particular, take a look at the gallery tests.

Loading your own test files

Test-agent loads your test files in an environment where they have access to mocha (among other bits of global state). One important test-agent global is require, which allows your test file to load a dependency (application code for instance). Sometimes, particularly if your application files expect to be loaded in a way that's different from test-agent's require, it can be useful to override test-agent's test file loader. You can do so in the following manner:

// setup.js
testAgentRuntime.testLoader = function(path) {
  // We're going to return a promise to represent
  // the loading state of the test file at this path.
  return new Promise((accept, reject) => {
    // Load the parameter test file and resolve (or reject) appropriately
    // ...
  });
};

Using mocks

TBD

Advanced: what does the script do?

Generating a profile

You need a profile that is generated by this command:

NO_LOCK_SCREEN=1 DEBUG=1 DESKTOP=0 make

This generates a debug profile in gaia/profile-debug, overriding a previous profile if you already have one.

• DEBUG=1 enables the httpd.js extension that makes it possible to directly use the files from the apps/ directory.
• NO_LOCK_SCREEN=1 disables the lock screen, which is necessary in B2G Desktop because it's not possible to unlock it using the mouse.
• DESKTOP=0 disables the other addons we normally use in DEBUG mode to run Gaia in Firefox.

Launch the WebSocket server

Test agent (the test runner) ships with a built in WebSocket server that lets you remotely message the browser or device to queue a test run. Often you will want to develop with time saving features like a file watcher that will run your tests when a test file or implementation changes. To take advantage of these features you need to start the server:

make test-agent-server

Using the WebSocket server provides other tools such as a command line reporter for test results (watch the terminal you ran the command from), a Growl reporter, syntax error notifications, and more.

The agent also watches for modifications in files, and automatically runs the associated tests. It runs when you save the test or if you save the tested file (we use the convention where the test filename is the tested filename with _test appended, see below for more examples). It watches only existing files so if you create a new file, you have to restart the agent.

Running unit tests with Firefox Nightly

You can launch Gaia in Firefox Nightly with the following commands:

cd <path to gaia>
<path to nightly>/firefox --no-remote -profile <path to gaia>/profile-debug/ http://test-agent.gaiamobile.org:8080/

You can use Firebug or the integrated debugger to debug the tests; use the debugger keyword to break in the debugger.

Running unit tests with B2G Desktop

Launch Gaia and start the "Test Agent" app. From the Test gent app you can select tests to run from the UI.

Advanced: how does the test-agent work?

The Test Agent lives in its own Github repository. You can have look there to understand how it works under the hood.

Toubleshooting

The test runner stays blank

Normally you'll see the list of existing tests displayed in the test runner (B2G Desktop, Firefox or Mulet). If this does not happen, this could be because something changed in Gecko. Usually we fix this by setting a pref, thus the easiest is to generate a new profile. You can do that by adding the option -f (for force profile generation) to bin/gaia-test:

bin/gaia-test -f
bin/gaia-test -d -f

Error: listen EADDRINUSE

This happens when a test runner already runs. You should look at your other terminal windows !

I want to debug the included httpd extension

You can launch the script with DEBUG=*:

DEBUG=* bin/gaia-test