So you’re interested in contributing to Mozilla Web QA automation projects but don’t know where to start? This doc will help you get up and running a set of tests locally.
We have a separate project for each web site that we test, so the first step is to choose a project. You can look at the list of our automation projects, or you could simply use the Addon-Tests project to start. Once you’ve decided which project you’d like to work on, follow this guide to run the existing tests.
You should join us on IRC at http://irc.mozilla.org, #mozwebqa and #qa. You can say hello there, and get help if you get stuck on these instructions.
All of our projects use Selenium/WebDriver and a fairly similar set of software requirements. However some projects may have specific requirements. If the any of these steps does not work as expected you may want to check out the README file in the project’s root to see if there’s anything special needed.
All of the following commands are performed from a terminal window (Linux/Mac) or shell (Windows).
Step 1: Install required software
In order to run and work on our tests you need both Git and Python installed. You will also need pip, which is a package manager for Python.
Excellent instructions for installing Git on all platforms can be found at Github.
Python can be downloaded from here. Select the appropriate version for your platform and run the installer, we recommend the highest version 2 package that is available.
You can verify that Python has been successfully installed by running the following command:
python --version
Windows only:
After installing Python, install SetupTools from here. Once that is done you’ll need to add the Python folder (e.g., C:\Python26) and the Python scripts folder (e.g., C:\Python26\Scripts) into your path via System Variables: System Properties->Environment Variables->System Variables-> Path.
Linux only:
If you are running on Ubuntu/Debian, after installing Python, install SetupTools via
sudo apt-get install python-setuptools
After Python is successfully installed, install pip by running the following command:
easy_install pip
You now have all of the software required to run Web QA tests!
Step 2: Clone the project
In order to simply run the tests for one of our projects, you'll first need to clone the repository on your machine. To do this simply visit the repo page for one of the automation projects and copy the URL.
git clone --recursive https://github.com/mozilla/Addon-Tests
Alternatively,
git clone --recursive git@github.com:mozilla/Addon-Tests.git
If you want to make any changes to tests, or add your own tests, you’ll need to fork the repository (repo) first, to obtain your own copy. In order to do this, click the fork button which you’ll find near the upper right-hand corner of the page:
You can then clone the forked repo (which means to create a working copy of all the code locally) to your machine. Switch to the folder on your machine in which you’d like to store the project, and issue the following command:
git clone --recursive git@github.com:[your username]/[project].git
For example:
git clone --recursive git@github.com:bobsilverberg/Addon-Tests.git
Pro tip: You can find the url to use to clone the project via the project’s Github page. Look for this area on the right-hand side of the page, and click on the little clipboard icon to copy the url to your clipboard:
If you don’t already have an SSH key set up to work with github, try clicking “https”, then copy the url for the repo to use in your git clone command.
Step 3: Install required Python packages
Each project has its own requirements.txt which lists Python packages needed for running the tests. In order to do that, you would first need to create or activate a Python virtual environment. While most of us have had some experience using virtual machines, virtualenv is something else entirely. It’s used to keep libraries that you install from clashing and messing up your local environment. If you already have a virtual environment for the desired tests, then you should activate it, otherwise you might want to create a new virtual environment. After installing virtualenv, installing virtualenvwrapper will give you some nice commands to use with virtualenv.
For a more detailed discussion of virtualenv and virtualenvwrapper, check out our quickstart guide and also this blog post.
Thereafter you can install the required packages using:
pip install -Ur requirements.txt
Step 4: Create a credentials file
Most of our projects include tests that need to log in to the web site. In order to do this we store the credentials for the site in either a credentials.yaml or a variable.json file. Depending on the project, you can find a sample credentials.yaml or a sample variable.json file in the root of the project. In order to run tests that need to log in, you will need to update this file with some valid credentials. First you’ll need a username and password for the site in question, which you can generally obtain by registering for the site, then you can add that information to the credentials file on your machine. Additionally, for testing payment related tests you could add paypal sandbox credentials.
If you do not see a sample credentials.yaml or a variable.json file in the root of the project then you can skip this step!
Pro tip: If you do not want Git to think that your credentials.yaml file has changed from the one in the project repo, create a new copy of the file in a different location and then point to it from the command line. For example, you can create a file in your home folder called “amo_credentials.yaml” and then point to that.
Step 5: Choose an execution environment
Most of the web sites we test have multiple environments, so before running the tests you need to choose where you want them to run. In general, a project will have one or more of the following environments available:
- Production – this is the “live” environment, the site which is actually visited by the site’s users. The url generally contains “mozilla.org” in it. For example, for addons it would be https://addons.mozilla.org.
- Staging – this is an environment where code lives after it has graduated from development but before going to production. The url generally contains “allizom.org” in it. For example, for addons it would be https://addons.allizom.org.
- Development – this is the environment that developers use to test their code. The url generally contains “dev-” in it. For example, for addons it would be https://addons-dev.allizom.org.
Pro tip: There is a file in the root of each project, called “mozwebqa.cfg” that contains some defaults for running the tests. It usually specifies a default environment for the tests using the “baseurl” setting. For example, mozwebqa.cfg in the addon-tests project contains the line:
baseurl = https://addons-dev.allizom.org
Step 6: Run the tests!
Once you have all the above in place, you are ready to start running some tests locally. Let’s give it a try: open a console and make sure you’re in the project’s directory.
To run all the tests:
py.test --driver=Firefox
To run all the tests with a credentials.yaml file:
py.test --driver=Firefox --credentials=/path/to/credentials.yaml
or in case of variables.json file:
py.test --driver=Firefox --variables=/path/to/variables.json
To run all of the tests in one file:
py.test --driver=Firefox tests/test_file.py
To run one test in a file:
py.test --driver=Firefox tests/test_file.py -k name_of_test_to_run
To run tests on an environment other than the default, specify the “–baseurl” option:
py.test --driver=Firefox --baseurl=https://addons.allizom.org
If you are running tests on the development or staging environments and was to run tests that change data (hint: you ususally do), you will need to also specify the “–destructive” command line option. For example:
py.test --driver=Firefox --baseurl=https://addons.allizom.org --destructive
The pytest-mozwebqa plugin has advanced command line options for reporting and using browsers. To see the options available, try running:
py.test --help
Also see the documentation on the plugin’s project page.
[Optional/Intermediate level]
A Selenium Grid can be used to run and Webdriver tests across many different browsers. The Web QA team also has a project which holds a Selenium Grid configuration for Windows and Mac OS.
Follow our quick start guide to get Selenium Grid up and running.
Now you’re ready to contribute!
If you’d like to learn more, the next post in this series is on Automated Tests: Analyzing Results.
Step 7: Get the Badge!
Once you have completed the steps above, you are eligible for the new Web QA badge! You can see the badge here:
https://badges.mozilla.org/en-US/badges/badge/Running–Web-QA-Automation
Log in using Persona to nominate yourself, or someone else who has Web QA tests up and running. Make sure to include your GitHub username or email in your profile. Congratulations!