TPS Tests

TPS consists of a Firefox extension of the same name, along with a python test runner. The python test runner will read a test file (in JavaScript format), setup a Firefox profile with the necessary extensions and preferences, then launch Firefox and pass the test file to the extension. The extension will read the test file and perform a series of actions specified therein, such as populating a set of bookmarks, syncing to the Sync server, making bookmark modifications, etc.

A test file may contain an arbitrary number of sections, each involving the same or different profiles, so that one test file may be used to test the effect of syncing and modifying a common set of data (from a single Sync account) over a series of different events and clients.

Set up an environment and run a test

Prerequisite: Have a firefox account.

Even if you have a personal one, for testing purposes you should create a new firefox account.
(this will require a valid email address)

Steps

  1. Get the source code

    Clone mozilla-central (choose your flavor):

    • hg clone hg.mozilla.org/mozilla-central
      or
    • git clone github.com/mozilla/gecko-dev

  2. cd into the tps folder

    cd testing/tps

  3. Create the environment

    I suggest the path to be outside of the mc source tree

    python create_venv.py --username=%EMAIL% --password=%PASSWORD% %PATH%

  4. Activate the environment

    source %PATH%/bin/activate

  5. Run some tests

    Note that the testfile is NOT a path, it should only be the filename from services/sync/tests/tps/

    runtps --testfile %TEST_FILE_NAME% --binary %FIREFOX_BINARY_PATH%
An example on OSX:
runtps --testfile test_sync.js --binary /Applications/FirefoxNightly.app/Contents/MacOS/firefox
 
 

Example test

The easiest way to understand this is to look at a simple TPS test file.

/*
 * The list of phases mapped to their corresponding profiles.  The object
 * here must be in strict JSON format, as it will get parsed by the Python
 * testrunner (no single quotes, extra comma's, etc).
 */
var phases = { "phase1": "profile1",
               "phase2": "profile2",
               "phase3": "profile1" };
/*
 * Bookmark asset lists: these define bookmarks that are used during the test
 */
// the initial list of bookmarks to be added to the browser
var bookmarks_initial = {
  "menu": [
    { uri: "http://www.google.com",
      title "google.com",
      changes: {
        title: "Google"
      }
    },
    { folder: "foldera" },
    { folder: "folderb" }
  ],
  "menu/foldera": [
    { uri: "http://www.yahoo.com",
      title: "testing Yahoo",
      changes: {
        location: "menu/folderb"
      }
    }
  ]
};
// the state of bookmarks after the first 'modify' action has been performed
// on them
var bookmarks_after_first_modify = {
  "menu": [
    { uri: "http://www.google.com",
      title "Google"
    },
    { folder: "foldera" },
    { folder: "folderb" }
  ],
  "menu/folderb": [
    { uri: "http://www.yahoo.com",
      title: "testing Yahoo"
    }
  ]
};
/*
 * Test phases
 */
Phase('phase1', [
  [Bookmarks.add, bookmarks_initial],
  [Sync, SYNC_WIPE_SERVER]
]);
Phase('phase2', [
  [Sync],
  [Bookmarks.verify, bookmarks_initial],
  [Bookmarks.modify, bookmarks_initial],
  [Bookmarks.verify, bookmarks_after_first_modify],
  [Sync]
]);
Phase('phase3', [
  [Sync],
  [Bookmarks.verify, bookmarks_after_first_modify]
]);

 

The effects of this test file will be:

  1. Firefox is launched with profile1, the TPS extension adds the two bookmarks specified in the bookmarks_initial array, then they are synced to the Sync server. The SYNC_WIPE_SERVER argument causes TPS to set the firstSync="wipeServer" pref, in case the Sync account already contains data. Firefox closes.
  2. Firefox is launched with profile2, and all data is synced from the Sync server. The TPS extension verifies that all bookmarks in the bookmarks_initial list are present. Then it modifies those bookmarks by applying the "changes" property to each of them. E.g., the title of the first bookmark is changed from "google.com" to "Google". Next, the changes are synced to the Sync server. Finally, Firefox closes.
  3. Firefox is launched with profile1 again, and data is synced from the Sync server. The TPS extension verifies that the bookmarks in bookmarks_after_first_modify list are present; i.e., all the changes performed in profile2 have successfully been synced to profile1. Lastly, Firefox closes and the tests ends.

Test format

TPS tests are written in JavaScript, with a particular syntax.  Each test has three sections:

  1. A list of test phases.
  2. A list of static or dynamically created asset lists, which contain the bookmarks, history entries, form data values, etc, that will be used by the test.
  3. A list of phases to execute.

Test phases

The test phase list looks like this:

/*
 * The list of phases mapped to their corresponding profiles.  The object
 * here must be in strict JSON format, as it will get parsed by the Python
 * testrunner (no single quotes, extra comma's, etc).
 */
var phases = { "phase1": "profile1",
               "phase2": "profile2",
               "phase3": "profile1",
               "phase4": "profile2" };

This section maps test phases to profiles.  The profile names are arbitrary; they will be generated by the TPS testrunner.  The phase names much match the names of the test phases defined in the phase section.  As noted in the comments above, this section must be in strict JSON format, as it will get parsed by the Python testrunner.

Asset lists

A test file will contain one or more asset lists, which are lists of bookmarks, passwords, or other types of browser data that are relevant to Sync. The format of these asset lists vary depending on asset type, and are explained in detail at these links:

Test Phases

The phase blocks are where the action happens! They tell TPS what to do. Each phase block contains the name of a phase, and a list of actions. TPS iterates through the phase blocks in alphanumeric order, and for each phase, it does the following:

  1. Launches Firefox with the profile from the phases object that corresponds to this test phase.
  2. Performs the specified actions in sequence.
  3. Determines if the phase passed or failed; if it passed, it continues to the next phase block and repeats the process.

A phase is defined by calling the Phase function with the name of the phase and a list of actions to perform:

Phase('phase1', [
  [Bookmarks.add, bookmarks_initial],
  [Passwords.add, passwords_initial],
  [History.add, history_initial],
  [Sync, SYNC_WIPE_SERVER],
]);

Each action is an array, the first member of which is a function reference to call, the other members of which are parameters to pass to the function.  Each type of asset list has a number of built-in functions you can call, described in the section on Asset lists; there are also some additional built-in functions.

Built-in functions

Sync(options)

Initiates a Sync operation.  If no options are passed, a default sync operation is performed.  Otherwise, a special sync can be performed if one of the following are passed:  SYNC_WIPE_SERVER, SYNC_WIPE_CLIENT, SYNC_RESET_CLIENT.

Logger.logInfo(msg)

Logs the given message to the TPS log.

Logger.AssertTrue(condition, msg)

Asserts that condition is true, otherwise an exception is thrown and the test fails.

Logger.AssertEqual(val1, val2, msg)

Asserts that val1 is equal to val2, otherwise an exception is thrown and the test fails.

Custom functions

You can also write your own functions to be called as actions.  For example, consider the first action in the phase above:

[Bookmarks.add, bookmarks_initial]

You could rewrite this as a custom function so as to add some custom logging:

[function() {
  Logger.logInfo("adding bookmarks_initial");
  Bookmarks.add(bookmarks_initial);
}]

Normally, custom functions are synchronous, but they can be made asynchronous as the following example shows:

  [function() {
    StartAsyncOperation();
    Logger.logInfo("waiting 5 seconds");
    Utils.namedTimer(function() {
      FinishAsyncOperation();
      Logger.logInfo("done waiting 5 seconds");
    }, 5000, TPS, "timer1");
  }]

Code calls StartAsyncOperation() to indicate the beginning of an asynchronous function.  Once called, the next action in the test phase will not be started until FinishAsyncOperation() is called.

Asset lists can be built using custom functions as well:

var hundred_bookmarks = { menu: [] };
for (var i = 1; i < 101; i++) {
  hundred_bookmarks.menu.push(
    {uri: 'http://www.google.com/search?q=highway' + i,
     title: 'Highway' + i + ' search results'
    });
}
Phase('phase1', [
  [Bookmarks.add, hundred_bookmarks],
  [Sync, SYNC_WIPE_SERVER]
]);

 

 

Mozmill tests in TPS

It's possible to run Mozmill tests from TPS.  You do this by specifying the name of the Mozmill test in the TPS test phase, like so:

Phase('phase1', [
  [RunMozmillTest, 'mozmill_sanity.js'],
  [Sync, SYNC_WIPE_SERVER]
]);
Phase('phase2', [
  [Sync],
  [RunMozmillTest, 'mozmill_sanity2.js'],
]);

The Mozmill tests should be in the same directory as the TPS test that calls them.  The version of Mozmill bundled with TPS is 2.0; the source here:  https://github.com/mozautomation/mozmill

To learn about writing Mozmill tests, see:

Because Mozmill tests which run under TPS are run using the TPS test runner and not the Mozmill one, they do not support python callbacks or restart tests (although you can achieve the same effect by running tests in different TPS test phases).

Firefox QA and the Thunderbird team each maintain a repository of Mozmill tests; they may be helpful as examples.  These can be found at:

Initiating a sync in a Mozmill TPS test

There are different ways of initiating a sync in a Mozmill TPS test.  One is to initiate the sync from the TPS "wrapper", for example:

Phase('phase2', [
    [Sync],
    [RunMozmillTest, 'mozmill_sanity2.js'],
]);

This works fine as long as you want to perform the sync before or after the Mozmill test is run.  If you want to perform the sync during the Mozmill test itself, you'll need to import sync.jsm in the Mozmill test, and then call TPS.Sync():

Components.utils.import('resource://tps/sync.jsm');
function test_something() {
  assert.equal(TPS.Sync(), 0, "sync succeeded");
}

sync.jsm provides the following functions to your TPS Mozmill tests:

TPS.Sync(options)

Initiates a sync operation.  Even though sync operations are asynchronous, TPS.Sync() is synchronous, and will not return until the sync operation has completed (successfully, or otherwise).   Returns 0 on success.  If its return is non-zero, it will be a string describing the failure.

TPS.Sync() accepts an optional parameter describing the type of sync to perform, which can be one of:  "wipe-client", "wipe-server", or "reset-client".  If none of these is specified, a regular sync is performed.

Calling TPS.Sync() will initiate a sync but will not set up any sync account.   Thus, if you call this without doing something else to set up the sync account, the sync will fail.  There are three ways you could set up a sync account prior to calling TPS.Sync():

  1. Call Sync in the TPS wrapper before calling RunMozmillTest.  This will set up the profile with the sync account details in your config.json file.
  2. Call TPS.SetupSyncAccount() in your Mozmill test before calling TPS.Sync().  This will set up the profile same as above.
  3. Use the Mozmill test to interact with the Create Sync Account UI and create a Sync account like a user would.

TPS.SetupSyncAccount()

Configures the profile's sync account according to the credentials in your TPS config.json file.  This function does not initiate a sync.  This function does not return a value.

Troubleshooting and debugging tips for writing TPS tests

  1. TPS evaluates the whole file in every phase, so any syntax error(s) in the file will get reported in phase 1, even though the error may not be in phase 1 itself.
  2. Inspect tps.log. When a tps test fails, the log is dumped to tps.log in the virtualenv
  3. Try adding a sync account in the config.json. Others might be using the tps default account, resulting in strange behavior during your tests.
  4. run test_sync.js. This test generally validates your tps setup.
  5. Comment out the goQuitApplication() calls in services/sync/tps/extensions/tps/modules/tps.jsm (remember to undo this later!). You will have to manually quit the browser at each phase, but you will be able to inspect the browser state manually.
  6. Inspect about:sync-log. Every sync should have a log and every item synced should have a record.

Document Tags and Contributors

Tags: 
 Contributors to this page: teoli, andrei.eftimie, ally@mozilla.com, Sheppy, jgriffin, gps
 Last updated by: andrei.eftimie,