The Marionette Test Runner is a Python test runner for Marionette tests. It is maintained as runtests.py, and the command-line options for this script are available via:
python runtests.py --help
See Running Tests for examples of running tests. The examples refer to mach, but you can use the same options with runtests.py (mach marionette-test just calls runtests.py)
Running tests against an emulator or device
To have Marionette launch an emulator for you and run tests on that emulator, you specify the --emulator option. The emulator will be closed when the tests are done.
To have Marionette connect to an actual device where a browser is running, you --address=ADDRESS, where ADDRESS is the host:port that Marionette can connect to. Marionette runs on port 2828 by default, but you'll have to setup port forwarding using adb so that the Marionette client can connect to port 2828 on the device.
adb forward tcp:2828 tcp:2828
To have Marionette connect to an emulator which you've already launched and then launch a browser instance for you, you specify both the --emulator and --address options.
Note that, currently, Marionette is available on Fennec debug builds.
Running tests against desktop builds
To have Marionette launch a desktop application such as Firefox, you specify the --binary option. The application will be closed when the tests are done. If you want to use a specific profile you can also specify the --profile option.
If you are testing against Fennec you should also specify the --app=fennec option to ensure the correct preferences are set.
To have Marionette connect to a desktop application which you've already launched, you must specify the --address option.
Specifying tests to run
To specify tests to run, you can specify one or more individual files, one or more directories, or a manifest file.
When specifying directories, Marionette will attempt to run all files contained therein beginning with test_ or browser_ and ending with .py or .js extensions.
When specifying a manifest file, you can also specify a --type=TYPE option. This controls what type of tests from the manifest are run. TYPE can be any combination of values listed in the manifest file, combined with + or - symbols. For instance, to run tests which are compatible with both the browser and b2g, you could specify --type=browser+b2g. To specify tests which are compatible with b2g and do not require an emulator, you could specify --type=b2g-qemu.
Test variables
The --testvars option allows you to specify a path to a JSON file with variables that can be used by your tests.
Examples
To run a specific test on a device which has local port 2828 forwarded to port 2828 on the device, use:
python runtests.py --address=localhost:2828 /path/to/test
Contribute to Marionette Test Runner
If you'd like to work on improving the Marionette Test Runner, including writing unit tests for it, visit our Developer Setup guide as well as the Marionette wiki