Mozilla automated testing

You've just written a feature and (hopefully!) want to test it. Or you've decided that an existing feature doesn't have enough tests and want to contribute some. But where do you start? You've looked around and found references to things like "xpcshell" or "web-platform-tests" or "talos". What code, features or platforms do they all test? Where do their feature sets overlap? In short, where should your new tests go? This document is a starting point for those who want to start to learn about Mozilla's automated testing tools and procedures. Below you'll find a short summary of each framework we use, and some questions to help you pick the framework(s) you need for your purposes.

Note: If you read this article and still have questions, try reading some of the further links provided below for more information, and hop on to the #ateam or #qa IRC channels or Mozilla Forums to ask questions.

In production

These tests are found within the mozilla-central tree, along with the product code. They are all run when a changeset is pushed to mozilla-central, mozilla-inbound, or try, with the results showing up on Treeherder. They can also be run on their own.

The tests are run on machines in a very large pool. For the most part, all tests of a particular type run on the same pool of machines, regardless of branch. One substantial exception is that try builds are performed on a pool isolated from all other builds.

Note: refer also to how test harnesses work for more information on how these tests are run.

Linting

Lint tests help to ensure better quality, less error-prone code by analysing the code with a linter.

Symbol	Name	File type	Platform		Process		Environment		Privilege		What is tested
Symbol	Name	File type	Desktop	Mobile	Parent	Child	Shell	Browser Profile	Low	High	What is tested
`ES`	ESLint	JS	Yes	Yes	N/A		Terminal	–	N/A		Javascript is analysed for correctness.
`f8`	flake8	Python	Yes	Yes	N/A		Terminal	-	N/A		Python analysed for style and correctness.
`W`	wpt lint	All	Yes	No	N/A		Terminal	–	N/A		web-platform-tests analysed for style and manifest correctness
`android-checkstyle`	android-checkstyle	Java	No	Yes	N/A		Terminal	-	N/A		Java analysed for coding style consistency.
`android-lint`	android-lint	Java	No	Yes	N/A		Terminal	-	N/A		Java analysed for common Android coding errors.
`android-findbugs`	android-findbugs	Java	No	Yes	N/A		Terminal	-	N/A		Java analysed for common Java coding errors.

Functional testing

Automated Test Suites
Symbol	Name	File type	Platform		Process		Environment		Privilege		What is tested
Symbol	Name	File type	Desktop	Mobile	Parent	Child	Shell	Browser Profile	Low	High	What is tested
`R(J)`	JS Reftest	JS	Yes	Yes	N/A		JSShell	–	N/A		The JavaScript engine's implementation of the JavaScript language.
`R(C)`	Crashtest	HTML XHTML SVG	Yes	Yes	–	Yes	Content	Yes	Yes	–	That pages load without crashing, asserting, or leaking.
`R(R)`	Reftest	HTML XHTML SVG	Yes	Yes	–	Yes	Content	Yes	Yes	–	That pages are rendered (and thus also layed out) correctly.
`R(Rs)`	Reftest sanity	?	–	–	?	?	?	?	?	?	?
`Cpp`	Compiled code	C++ Python	Yes	Yes	N/A		Terminal	–	N/A		Code not exposed to JavaScript (using C++ executables), and the state of the source tree (using Python scripts).
`X`	xpcshell	JS	Yes	Yes	Yes	Allow	XPCShell [Note 1]	Allow	–	Yes	Low-level code exposed to JavaScript, such as XPCOM components.
`M(oth)`	IPC	?	–	–	?	?	?	?	?	?	Plugin APIs, particularly out-of-process plugins.
`M(oth)`	Accessibility (mochitest-a11y)	?	Yes	–	?	Yes	Content	Yes	?	?	Accessibility interfaces.
`M(1) M(2) M(…)`	Mochitest plain	HTML XHTML XUL	Yes	Yes	–	Yes	Content	Yes	Yes	Allow	Features exposed to JavaScript in web content, like DOM and other Web APIs, where the APIs do not require elevated permissions to test.
`M(oth)`	Mochitest chrome	HTML XHTML XUL	Yes	–	Allow	Yes	Content	Yes	–	Yes	Code requiring UI or JavaScript interactions with privileged objects.
`M(bc)`	Mochitest browser-chrome	JS	Yes	–	Yes	Allow	Browser	Yes	–	Yes	How the browser UI interacts with itself and with content.
`M(dt)`	Mochitest devtools	JS	Yes	–	Yes	Allow	Browser	Yes	–	Yes	Firefox developer tools. Based on Mochitest browser-chrome.
`M(rc)`	Mochitest robocop	Java	–	Yes	?	?	?	?	?	?	Native UI of Android devices (sends events to the front end). Some use a combination of Java and JavaScript.
`M(gl)`	Mochitest WebGL	?	?	?	?	?	?	?	?	?	?
SM(...) SM(pkg)	SpiderMonkey automation	JS C++	Yes	No	N/A		JSShell	-	Yes	-	SpiderMonkey engine shell tests and JSAPI tests
`W`	web-platform-tests	HTML XHTML	Yes	–	–	Yes	Content	Yes	Yes	–	Standardized features exposed to ECMAScript in web content; tests are shared with other vendors.
`Wr`	web-platform-tests reftests	HTML XHTML	Yes	–	–	Yes	Content	Yes	Yes	–	Layout and graphic correctness for standardized features; tests are shared with other vendors
`Mn`	Marionette	Python	Yes	–	?	?	Content Browser	?	–	Yes	Large out-of-process function integration tests and tests that do communication with multiple remote Gecko processes.
`Mn-h`	Marionette harness tests	Python	-	-	-	-	-	-	-	-	Marionette Python Runner
`JP`	Jetpack	JS	?	?	?	?	?	?	?	?	Add-on SDK code included in Firefox. If you need help interpreting test failures or fixing bugs in the SDK code, talk to the SDK team in #jetpack.
`V`	Valgrind	?	Linux only	–	?	?	?	?	?	?	Memory-related errors, such as heap block overflows/underflows, use of uninitialized memory, bad frees, and memory leaks.
`VP`	VideoPuppeteer External Media Tests	Python	Yes	-	-	Yes	Content	Allow	Yes	-	MSE and EME video playback on sites such as YouTube
Fxfn Fxup	Firefox UI Tests	Python	Yes	-	?	?	Content Browser	Yes	-	Yes	Integration tests with a focus on the user interface and localization.
T-hn	Telemetry Test Harness	Python	Yes	No	?	?	Content Browser	Yes	-	Yes	Integration tests with a focus on telemetry client.
A(...)	Autophone	HTML	-	Yes	?	?	Content	Yes	?	?	Performance and select Unit Tests on Android devices.
`android-test`	android-test	Java	-	Yes	N/A	N/A	Java	No	N/A	N/A	Run Android local unit tests.

Retired Automated Test Suites
Symbol	Name	File type	Platform			Process		Environment		Privilege		What is tested
Symbol	Name	File type	Desktop	Mobile	B2G	Parent	Child	Shell	Browser Profile	Low	High	What is tested
`Gb`	gaia-build-integration	?	–	–	Yes	?	?	?	?	?	?	The Gaia build system.
`Gip`	gaia-ui-tests	Python	–	–	Yes	?	?	?	?	?	?	Gaia integration. Marionette- and Python- based.
`Gij`	gaia-integration	JS	–	–	Yes	?	?	?	?	?	?	Gaia integration. Marionette- and JavaScript- based.
`Li`	gaia-linter	?	–	–	Yes	?	?	?	?	?	?	Gaia JavaScript code formatting.
`Mnw`	Marionette WebAPI	JS	–	–	Emul. only	?	?	?	?	?	?	WebAPIs. This takes advantage of the Firefox OS emulator's ability to virtualize much of the hardware on a B2G device.

Table key

Symbol

Abbreviation for the test suite used by Treeherder. The first letter generally indicates which of the test harnesses is used to execute the test. The letter in parentheses identifies the actual test suite.

Name

Common name used when referring to the test suite.

File type

When adding a new test, you will generally create a file of this type in the source tree and then declare it in a manifest or makefile.

Platform

Most test suites are supported only on a subset of the available plaforms and operating systems. Unless otherwise noted:

Desktop tests run on Windows, Mac OS X, and Linux.
Mobile tests run on Android emulators or remotely on Android devices.

Process

When Parent is indicated, the test file will always run in the parent process, even when the browser is running in Electrolysis (e10s) mode.
When Child is indicated, the test file will run in the child process when the browser is running in Electrolysis (e10s) mode.
The Allow label indicates that the test has access to mechanisms to run code in the other process.

Environment

The JSShell and XPCShell environments are limited JavaScript execution environments with no windows or user interface (note however that XPCShell tests on Android are run within a browser window.)
The Java environment is a limited Java execution environments with no user interface or connection to the Gecko rendering or JavaScript environment.
The Content indication means that the test is run inside a content page loaded by a browser window.
The Browser indication means that the test is loaded in the JavaScript context of a browser XUL window.
The Browser Profile column indicates whether a browser profile is loaded when the test starts. The Allow label means that the test can optionally load a profile using a special command.

Privilege

Indicates whether the tests normally run with low (content) or high (chrome) JavaScript privileges. The Allow label means that the test can optionally run code in a privileged environment using a special command.

Performance testing

Talos is the framework used for performance testing.

Performance Test Suites
Symbol	Name	Platform		What is tested
Symbol	Name	Desktop	Mobile	What is tested
`T(tp)`	tp	Yes	Yes	The load time of a set of test web pages taken from the Alexa top 500, with the full UI enabled.
`T(s)`	svg	Yes	Yes	SVG rendering time.
`T(c)`	chrome	Yes	-	Various suites with the full UI enabled.
`T(di)`	dirty	Yes	-	A populated history database (places.sqlite) that more closely resembles the typical viewing habits of an average user.
`T(dr)`	Dromaeo	Yes	-	JavaScript performance tests.
`T(n)`	nochrome	Yes	-	Various suites with the full UI disabled.
`T(tpn)`	tp nochrome	-	Yes	The load time of a set of test web pages taken from the Alexa top 500, with the full UI disabled.
`T(tsp)`	tspaint	-	Yes	The startup time of Firefox, by opening the browser 20 times.
`T(s)`	tsvgx	-	Yes	SVG rendering time.
`T(cs)`	tcanvasmark	-	Yes	Canvas animation, using the third-party CanvasMark benchmark.
`T(rck2)`	tcheckerboard2	-	Yes	The amount of checkerboarding (delayed painting), by loading a test page then zooming and panning. Robocop-based.

So which should I use?

Generally, you should pick the lowest-level framework that you can. If you are testing JavaScript but don't need a window, use XPCShell or even JSShell. If you're testing page layout, try to use Reftest. The advantage in lower level testing is that you don't drag in a lot of other components that might have their own problems, so you can home in quickly on any bugs in what you are specifically testing.

Here's a series of questions to ask about your work when you want to write some tests.

Is it low-level code?

If the functionality is exposed to JavaScript, and you don't need a window, consider XPCShell. If not, you'll probably have to use compiled-code tests, which can test pretty much anything but are difficult to write properly and are often less suitable than a domain-specific harness. In general, this should be your last option for a new test, unless you have to test something that is not exposed to JavaScript.

Does it cause a crash?

If you've found pages that crash Firefox, add a crashtest to make sure future versions don't experience this crash (assertion or leak) again. Note that this may lead to more tests once the core problem is found.

Is it a layout/graphics feature?

Reftest is your best bet, if possible.

Do you need to verify performance?

Use Talos!

Are you testing UI features?

If it's mobile UI, look into Robocop. For desktop, try Marionette (specifically firefox-ui-tests if the page object model aka firefox-puppeteer should be used) or browser chrome tests (soon to be split out of Mochitest).

Are you testing Mobile/Android?

For Mobile UI, look at Robocop. For Mobile features that are purely Java, look at android-test. There are some specific features that Mochitest or Reftest can cover. Browser-chrome tests do not run on Android. If you want to test performance, Talos runs fine with a few limitations (use the --noChrome options) and smaller cycles (e.g. 10 iterations instead of 20, etc.)

Are you doing none of the above?

To get your tests run through buildbot, try Mochitest, or, if higher privileges are required, try Mochitest chrome tests.
For Desktop Firefox, or if you just want to see the future of Gecko testing, look into the on-going Marionette project.

Need to get more data out of your tests?

Most test jobs now expose an environment variable named $MOZ_UPLOAD_DIR. If this variable is set during automated test runs, you can drop additional files into this directory, and they will be uploaded to a web server when the test finishes. The URLs to retrieve the files will be output in the test log.

Need to set preferences for test-suites?

Sadly, the locations for preferences are all over the place, some are listed below - bug 1023483 has more details and locations.

profile.py generates Firefox & Thunderbird specific profiles:
- this affects at least reftest, xpcshell, mochitest & marionette
- https://hg.mozilla.org/mozilla-central/annotate/caf25344f73e/testing/mozbase/mozprofile/mozprofile/profile.py#l330
mochitests: http://dxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js
xpcshell: http://dxr.mozilla.org/mozilla-central/source/testing/xpcshell/head.js
Talos:
- global: https://hg.mozilla.org/build/talos/file/tip/talos/PerfConfigurator.py#l251
- per-test in test.py, e.g.: https://hg.mozilla.org/build/talos/file/49b74c08dad4/talos/test.py#l209
- the current Talos revision is pointed to here: testing/talos/talos.json#l8
reftests:
JS engine tests: https://dxr.mozilla.org/mozilla-central/source/js/src/tests/user.js

Note: The previous page was largely out of date. The new page is intended as a guide to the testing frameworks at Mozilla; some of the old page should be split out to a "Running Automated Tests" page to complement the "Developing Tests" page (which itself needs to be updated).

Also see the "Continuous Integration" page.