External Media Tests

The external-media-tests is a Marionette-based test suite designed to test playback of video elements on arbitrary web pages. Right now, tests using this framework play videos on YouTube and Netflix pages, but the basic tests should be usable for any website which plays back HTML5 video. MediaTestCase uses the Firefox Puppeteer library.

Information about the use of external-media-tests in Mozilla automation can be found on wiki.mozilla.org.

Setup

Normally, you get this source by cloning a Firefox repo such as mozilla-central. The path to these tests would be in <mozilla-central>/dom/media/test/external, and these instructions refer to this path as $PROJECT_HOME. You can also get these tests by downloading one of the tests.comon.zip files matching a downloaded Firefox build from http://archive.mozilla.org, or using mozdownload with the --extension tests.common.zip parameters.

Running from a build

If you have built Firefox using ./mach build from a source tree such as mozilla-central, you can run the following command:

$ ./mach external-media-tests

You can pass any of the test options on this command line. They are listed below.

Running with an installer and a tests payload

If you are testing a version of Firefox that you have not built, you must set up a virtualenv to run tests from. You will need a path to the installer or binary of Firefox.

Create a virtualenv called foo.

$ virtualenv foo 
$ source foo/bin/activate #or `foo\Scripts\activate` on Windows

Install external-media-tests in development mode. (To get an environment that is closer to what is used in Mozilla's automation jobs, run pip install -r requirements.txt first.)

$ python setup.py develop ``` Now `external-media-tests` should be a recognized command. Try `external-media-tests --help` to see if it works.

Running the Tests

In the examples below, $FF_PATH is a path to a recent Firefox binary. If you are running from a source build, the commands below should be invoked with:

$ ./mach external-media-tests

If you are running with a virtualenv, you will need to run like this:

$ external-media-tests --binary $FF_PATH

$ external-media-tests --installer $FF_INSTALLER_PATH

$ external-media-tests --installer-url <url to installer package>

The following examples assume that you will use one of these command lines instead of $EXTERNAL-MEDIA-TESTS. This runs all the tests listed in $PROJECT_HOME/external_media_tests/manifest.ini:

$ $EXTERNAL-MEDIA-TESTS

You can also run all the tests at a particular path:

$ $EXTERNAL-MEDIA-TESTS some/path/foo

Or you can run the tests that are listed in a manifest file of your choice:

$ $EXTERNAL-MEDIA-TESTS some/other/path/manifest.ini

By default, the URLs listed in external_media_tests/urls/default.ini are used for the tests, but you can also supply your ini file of URLs:

$ $EXTERNAL-MEDIA-TESTS --urls some/other/path/my_urls.ini

Running EME Tests

If you are running EME tests in a local build, you must build with the following parameters in your mozconfig file before you build Firefox:

ac_add_options --enable-eme=adobe

You must also copy the files plugin-container.exe and voucher.bin from a production build into your <objdir>/dist/bin directory. This is not necessary when running production builds.

To run EME tests, you must use a Firefox profile that has the credentials of the service you are using stored in it. With Netflix, this will be created when you log in and save the credentials. You must also use a custom .ini file for URLs to the provider's content and indicate which test to run, as above. For example:

$ $EXTERNAL-MEDIA-TESTS some/path/tests.ini --profile custom_profile_dir --urls some/path/provider-urls.ini

Note, the profile argument needs to specify the directory of a profile, not the name of that profile. For example, the following arguments could be used:

$ $EXTERNAL-MEDIA-TESTS dom/media/test/external/external_media_tests/playback/eme.ini --profile ~/AppData/Roaming/Mozilla/Firefox/profiles/6usy8n4x.mozilla-debug-netflix/ --urls dom/
media/test/external/external_media_tests/urls/netflix/default.ini

Running tests in a way that provides information about a crash

What if Firefox crashes during a test run? You want to know why! To report crash data, the test runner needs access to a minidump_stackwalk binary and a symbols.zip file.

Download a minidump_stackwalk binary for your platform (save it wherever). Get it from Breakpad.
Make minidump_stackwalk executable:

$ chmod +x path/to/minidump_stackwalk # Linux and Mac

Create an environment variable called MINIDUMP_STACKWALK that points to that local path:

$ export MINIDUMP_STACKWALK=path/to/minidump_stackwalk

Download the crashreporter-symbols.zip file for the Firefox build you are testing and extract it. For example, ftp://ftp.mozilla.org/pub/firefox/tinderbox-builds/mozilla-aurora-win32/1427442016/firefox-38.0a2.en-US.win32.crashreporter-symbols.zip
Run the tests with a --symbols-path flag:

$ $EXTERNAL-MEDIA-TESTS --symbols-path path/to/example/firefox-38.0a2.en-US.win32.crashreporter-symbols

To check whether the above setup is working for you, trigger a (silly) Firefox crash while the tests are running. One way to do this is with the crashme add-on: you can add this to Firefox even while the tests are running. Another way on Linux and Mac OS systems:

Find the process id (PID) of the Firefox process being used by the tests.

$ ps x | grep 'Firefox'

Kill the Firefox process with SIGABRT.

# 1234 is an example of a PID
$ kill -6 1234

Somewhere in the output produced by $EXTERNAL-MEDIA-TESTS, you should see something like:

0:12.68 CRASH: MainThread pid:1234. Test:test_basic_playback.py TestVideoPlayback.test_playback_starts.
Minidump anaylsed:False.
Signature:[@ XUL + 0x2a65900]
Crash dump filename:
/var/folders/5k/xmn_fndx0qs2jcpcwhzl86wm0000gn/T/tmpB4Bolj
.mozrunner/minidumps/DA3BB025-8302-4F96-8DF3-A97E424C877A.dmp
Operating system: Mac OS X
                  10.10.2 14C1514
CPU: amd64
     family 6 model 69 stepping 1
     4 CPUs
Crash reason: EXC_SOFTWARE / SIGABRT
Crash address: 0x104616900
...

Setting up for network shaping tests (browsermobproxy)

Download the browsermob proxy zip file from http://bmp.lightbody.net/. The most current version as of this writing is browsermob-proxy-2.1.0-beta-2-bin.zip.
Unpack the .zip file.
Verify that you can launch browsermobproxy on your machine by running <browsermob>/bin/browsermob-proxy (or browsermob-proxy.bat on Windows) on your machine. You may have to do a lot of work to install and use a Java that browsermobproxy would like.
Import the certificate into your Firefox profile. Select Preferences->Advanced->Certificates->View Certificates->Import... Navigate to <browsermob>/ssl-support and select cybervilliansCA.cer. Select all of the checkboxes.
Tell Marionette where browsermobproxy is and what port to start it on. Add the following command-line parameters to your external-media-tests command line:

--browsermob-script <browsermob>/bin/browsermob-proxy --browsermob-port 999 --profile <your saved profile>

You can then call browsermob to shape the network. You can find an example in external_media_tests/playback/test_playback_limiting_bandwidth.py. Another example can be found at https://dxr.mozilla.org/mozilla-central/source/testing/marionette/harness/marionette/tests/unit/test_browsermobproxy.py.

A warning about video URLs

The ini files in external_media_tests/urls may contain URLs pulled from Firefox crash or bug data. Automated tests don't care about video content, but you might: visit these at your own risk and be aware that they may be NSFW. We do ever not intend to moderate or filter these URLs.

Writing a test

Tests go into $PROJECT_HOME/external_media_tests/playback.
Your test should inherit from MediaTestCase (or a subclass). You may also need to inherit from VideoPlaybackTestsMixin, which gives you test_playback_starts() and test_video_playback_partial().
You can check $PROJECT_HOME/external_media_tests/media_utils for other useful methods and tests to inherit.
If you have specific URLs you wish to test, create a .ini file and place it in $PROJECT_HOME/external_media_tests/urls, and pass it to your test via the '--urls' command-line parameter.
If you wish to create a suite of tests for a particular site or feature, you can list the tests you want into a .ini file in the $PROJECT_HOME/external_media_tests/playback folder, and pass that file in your test command-line.

API testing for the classes in external-media-tests are on readthedocs.org.

Generating API documentation

API documentation is generated from the python source code itself. The documentation is added with docstring; the resulting documentation is generated with the Sphinx tool.

The document structure is located in $PROJECT_HOME/docs. The .rst files are written for Sphinx to process. Please consult the Sphinx documentation if you need to add more sections to the .rst files we already have.

If you make changes to the docstrings in the python code, or if you make changes to the .rst files, you need to test your changes and see how they look. Follow these steps:

Setup a virtualenv and activate it.

virtualenv docenv
. docenv/bin/activate

Install Sphinx into your virtualenv.

pip install sphinx

Install External Media Tests into your virtualenv.

cd $PROJECT_HOME 
python setup.py install

Run the command to generate the docs

cd $PROJECT_HOME/docs
make html

Examing the resulting html files. The top-level html file will be at $PROJECT_HOME/docs/_build/html/index.html.

Note that your virtualenv must be active to rerun the make command. Note also that if you change your python code to change documentation, you must rerun python setup.py install to pick those changes up.

License

This software is licensed under the Mozilla Public License 2.0