This is an add-on tool on top of python marionette test to use imagemagick console utility 'compare' to compare device-captured screenshots for their validity. One should be familiar with running python marionette tests on device. For more information, visit here.
Installation Guide
-
Install ImageMagick 6.8.8.9. Check the console recognizes 'compare' command. It should show something like this:
Version: ImageMagick 6.8.8-9 Q16 x86_64 2014-03-23 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2014 ImageMagick Studio LLC
Features: DPC Modules
Delegates: bzlib freetype jng jpeg ltdl png xml zlib - Install PIL (Python Imaging Library).
sudo pip install Pillow - Get the latest gaia tree at https://github.com/mozilla-b2g/gaia
- Go to gaia/tests/python/gaia-ui-tests folder
- (Optional) start the virtual environment to run the gaiatest. Refer to this wiki.
- Connect a flame device via usb port. Make sure the developer option is enabled, as well as the 'Debugging via USB' with 'ADB' or 'ADB and DevTools'
- run command: adb forward tcp:2828 tcp:2828 to connect to the marionette server in the device
Usage Guide
Capturing & Analyzing Images
You have to first understand how the tool works. All helper methods are located in gaia-ui-tests/gaiatest/gaia_graphics_test.py file. The command-line options are defined in gaia-ui-tests/gaiatest/mixins/imagecompare.py
On the initial run, --store-reference-image command line option should be specified. When the script is being executed, marionette collects screenshots being captured by self.marionette.screenshot call. It captures the screenshots as the user would see in that instance. Then, the the top status bar of the captured screenshots are cropped, and renamed to show the ordering, test name, and device. After that, it is copied to the 'reference_images' folder under gaia-ui-tests folder by default. (The folder name can be changed with --reference_path <folder name relative to current location> command line option.) The user can visually inspect the images to verify they are proper, and decide they should be compared against all subsequent test runs for the specified device.
Now, for the subsequent runs without --store-reference-image command line option will enable the marionette to save the captured screenshots into 'screenshots' folder with status bar cropped as well. (The folder name can be changed with --screenshots_path <folder name relative to current location>.) By comparing the test name and the order number, the image_compare method inside the gaia_graphics_test.py file compares the corresponding reference image and the current screenshot image to generate a comparison result. The last parameter of image_compare method is the fuzz factor, which makes the imagemagick ignore minor pixel color differences. (By default, this is set to 10 percent.) This can be changed with the command line option --fuzz_factor <value>
In summary, following would be the typical procedure to follow:
- Make sure the --store-reference-image command line option is set.
- Example command: gaiatest --address=localhost:2828 --testvars=gaiatest/b2g-7.json --restart --type=b2g -- store-reference-image gaiatest/tests/graphics/<test or manifest name>
- go to the reference_images folder, and check they are in good shape.
- Remove the --store-reference-image command line option from step 2, and execute the same gaiatest command.
- Watch the result. If comparison test fails, go to the shots folder for diff image outputs.
Simulating UI Action
GaiaImageCompareTestCase class contains three methods to simulate UI action, and they can be used in conjunction with the invoke_screen_capture:
- edge_scroll(self,frame, direction, dist, release=True): this method simulates the user action of edge scroll to switch between apps. If release is set to False, then the edge scroll will stop without the finger release action. The script can be written to take screenshots in between, and call the release on the action object it returns.
- pinch(self, locator, zoom, duration): This method uses the pinch method in marionette.gestures. The user can script zoom in/out action with this method. Refer to gaiatest/tests/graphics/test_orientation_zoom.py script for example.
- scroll(self, locator, direction, distance, increments=None): this method simulates scroll up/down UI action. Verified to work in browser and gallery app. Refer to gaiatest/tests/graphics/test_orientation_zoom.py script for example.
Example Scripts
Currently all ImageCompare library enabled scripts are located in gaia-ui-tests/gaiatest/tests/graphics folder.
Automated Test Environment
Currently the imagecompare test suite executes upon every new nightly release in Jenkins, and you can see the result here: http://jenkins1.qa.scl3.mozilla.com/view/Graphics/
There are two jobs, a reference job and a smoketest job. reference job is executed manually, and after the visual inspection of created reference images, the build is marked to 'keep forever' in Jenkins. (Note that there should be only one build that is marked 'Keep Forever') Then, when the smoketest job is triggered by the completion of the download job of mozilla-central, (http://jenkins1.qa.scl3.mozilla.com/view/Downloads/job/flame-kk.mozilla-central.download/) it will use the artifacts from the build that is marked as 'keep forever' as the reference image.
Running Debug Builds and Analyzing logcat
Another type of jenkins imagecompare job is adhoc job.they are http://jenkins1.qa.scl3.mozilla.com/view/Graphics/job/flame-kk.mozilla-central.nightly.ui.graphics.adhoc.bitbar/ and http://jenkins1.qa.scl3.mozilla.com/view/Graphics/job/flame-kk.mozilla-central.nightly.ui.graphics.reference.adhoc.bitbar/ . These jobs are designed to run any scripts that are either in gaia repo, or in a personal branch. These adhoc jobs can be used to execute tests on a debug build as well, by specifying the location of the debug build within the pvtbuild repo. When running the tests on a debug build, the memory of the flame device should be set to 512MB or higher in the build parameter section, otherwise some of the scripts will not execute successfully.
Once the test is completed, the user can download the logcat.txt from the Workspace (tests/python/gaia-ui-tests) or Build Artifacts folder. Following grep commands should be useful to analyze the logcat for any debug assertion statements.
- grep -i 'INFO: TEST-START\|assert' logcat.txt > output.txt : This command will filter all assertion outputs along with the names of the test script, thus making it easier to identify in which script the assertion has occurred. The word 'assert' can be replaced with any other words.
- split -p TEST-START <filename> : This command will split the logcat into several files, one for each test script.