Firefox UI tests

The Firefox UI test suite is an automated testing framework built on top of Marionette for executing ui-level and end-to-end tests for Firefox Desktop. It can be used for tests which cannot be covered in other test suites like browser chrome tests due to restrictions or feature limitations (e.g. requirements for a restart of the browser). Compared to other test suites the Firefox UI tests can also be executed for localized builds of Firefox given that all interactions are done via UI elements retrieved via locale independent DOM properties. To ease the creation and maintainance of tests the Firefox puppeteer POM (page object model) has been created, which contains numerous wrapper classes for the most often used UI elements and API calls. And last but not least the tests are written sequentially, which will drop the complexity of all the asynchronicity in Firefox and allow even newcomers to get started.

Running the Firefox UI tests

To run the tests, first build Mozilla with your changes; then run

./mach firefox-ui-functional

This will launch your build, execute all existing tests, and report the results to stdout.

It is possible to run specific groups of tests. As with Marionette, the path given as an argument is the path to a test or directory within the Mozilla source tree. If the path points to a directory, then the tests in that directory and all of its subdirectories will be run.

For example, to run the tests in testing/firefox-ui/tests/functional/security the command would be:

./mach firefox-ui-functional testing/firefox-ui/tests/functional/security/

Run ./mach help firefox-ui-test for more options.

Type of Firefox UI tests

Currently the Firefox UI tests are divided into two separate types. This distinction has been made to clearly separate different kinds of tests, which have e.g. completely different requirements or purposes.

Functional tests

The functional tests, which are the general type of tests and are used for testing the behavior of Firefox as described above. Tests are using local or remote testcase data and are tagged appropriately in the corresponding manifest.ini file. This type of test covers nearly all existing tests and should be used for new tests.

Update tests

The update tests are a specific subset of the functional tests and are used to properly test the update procedure of Firefox. The tests check what the users are usually doing by upgrading e.g. Firefox 44.0 to Firefox 45.0. So these tests are very important for the release process and the QA signoff for new releases. That also means when comparing the tests with the functional tests two different Firefox builds are involved here. Those builds can also be some versions apart, and would require full backward compatibility.

Writing Firefox UI tests

Firefox UI tests support test cases written in Python, using Python's unittest framework. Tests are written as a subclass of one of the following testcase classes:

• MarionetteTestCase (The very basic testcase class which comes via Marionette itself)
• FirefoxTestCase (Extended testcase class for the use of firefox-puppeteer)
• UpdateTestCase (Extended testcase class on top of FirefoxTestCase for update tests)

Whereby individual tests belonging to instance methods that have a name starting with `test`. Additionally, you can provide setUp and tearDown instance methods to execute code before and/or after tests in a particular test class have run, but remember to call setUp/tearDown methods of the base testcase class in your provided function, since the base classes handle setup/teardown between test cases.

This test structure is illustrated here:

from firefox_puppeteer.testcases import FirefoxTestCase
class TestFoobar(FirefoxTestCase):
    def setUp(self):
        FirefoxTestCase.setUp(self)
        # code to execute before any tests are run
    def tearDown(self):
        # code to execute before any tests are run
        FirefoxTestCase.tearDown(self)
    def test_foo(self):
        # run test for 'foo'
    def test_bar(self):
        # run test for 'bar'

Further documentation in how to get started and which rules to obey when contributing to the Firefox UI test suite can be found in the A-Team bootcamp.

Adding a new Firefox UI test to the tree

To add a new Firefox UI test to the tree, add it to the manifest.ini file in the same folder as the test. Also remember that the test file's name must begin with "test_" for the test to be recognized as a Firefox UI test. If you are adding the first tests in a directory, make sure to also create a new manifest.ini file and include it in the manifest.ini of the parent folder.

Resource files

Some of the tests make use of local test data as served via a local webserver (we use wptserve). If the new test requires such a file please store it in testing/firefox-ui/resources/. It's URL can then be retrieved in the test via `self.marionette.absolute_url(%filename%)`.

How to disable failing tests

If test failures are getting reported to Treeherder the responsible persons of the specific test has to fix the failure. If a patch cannot be done in a short time and the test constantly fails, it has to be disabled aka marked as skipped. This can be done in two ways:

1. If all test methods within a single test file are failing the complete file has to be marked as disabled via the manifest file of the same folder. This can be done by using the `disabled` key followed by a description which explains why the test has been disabled. It should also contain a bug number for reference:
   
       [test_access_locationbar.py]
       skip-if = true  # Bug 1168727
    
2. In case of only one failing test method the above way would not be useful. Instead the one failing test method can be marked as skipped via the `unittest` module, or if available via a Marionette decorator (e.g. `@skip_if_e10s`):
   
       import unittest
       [..]
       @unittest.skip('Bug 123456')
       def testCheckAboutPrivateBrowsing(self):
   
       from marionette.marionette_test import skip_if_e10s

       [..]
       @skip_if_e10s('Bug 123456')
       def testCheckAboutPrivateBrowsing(self):

Other information

• Project documentation with current projects, contacts, and tips in how to contribute can be found on wiki.mozilla.org.