Firefox OS performance testing

Rendering and startup performance

Firefox OS improves startup experience in several ways :

• Each web application has its own instance of the Gecko rendering engine. Starting up this large, complicated engine is not free; because of that, Firefox OS keeps around a preallocated copy of the engine in memory. When an app starts up, it takes over this preallocated copy and immediately begins loading its application resources.
• Applications "start" more quickly when they're already running. To this end, Firefox OS tries to keep as many applications as running in the background as possible, while not regressing the user experience in foreground applications. This is implemented by intelligently prioritizing applications, and discarding background applications according to their priorities when memory is low. For example, it's more disruptive to a user if his\her currently-playing music player is discarded in the background, while their background calculator application keeps running. So, the music player is prioritized above the calculator automatically by Firefox OS and the calculator is discarded first when memory is low.
• Firefox OS prevents applications that are running in the background from impacting the user experience of foreground applications through two mechanisms. First, timers created by background apps are "throttled" to run at a low frequency. Second, background applications are given a low CPU priority, so that foreground applications can get CPU time when they need it.

In addition, Firefox OS includes several features designed to improve power usage that are common to mobile operating systems. The Firefox OS kernel will eagerly suspend the device for minimal power usage when the device is idle. Relatedly, ICs like the GPU, cellular radio, and Wifi radio are powered down when not being actively used. Firefox OS also takes advantage of hardware support for media decoding.

Memory and power usage

Firefox OS tries to keep as many applications running simultaneously as it can, but does have to discard applications sometimes, usually when the device runs out of memory.

Improving memory and power usage is a similar problem to speeding up the startup. Don't do unnecessary work, use efficient data structures, lazily load uncommonly-used UI resources, and ensure resources like images are optimized well. Modern CPUs can enter a lower-power mode when mostly idle. Applications that constantly fire timers or keep unnecessary animations running prevent CPUs from entering low-power mode. Power-efficient applications shouldn't do that.

When applications are sent to the background, a visibilitychange event is fired on their documents. This event is a developer's friend; applications should listen for it. Applications that drop as many loaded resources as possible when sent to the background will use less memory (and be less likely to be discarded, in the case of Firefox OS, see the note). This in turn means they will "start up" faster (by virtue of already being running) and have better UPP.

Firefox OS applications should prepare for the case when they  are  discarded. To improve user-perceived memory usage, that is to say, making the user feel that more of their state is being preserved, applications should save the state needed to return the current view if discarded. If the user is editing an entry, for example, the state of the edit should be saved when entering the background.

Measuring performance and diagnosing problems on Firefox OS

Before beginning to measure and diagnose performance problems, remember this:

A great strength of the web platform is that the same code written for "desktop" web browsers runs on Firefox OS on mobile devices. Developers should use this to improve productivity as much as possible.They should also develop the same on "desktops" in comfortable and productive environments. When the time comes to test performance, mobile devices must be used. Modern desktops can be more than 100x more powerful than mobile hardware. The lower-end the mobile hardware tested on, the better.

With that caveat, the sections below describe tools and processes for measuring and diagnosing performance issues.

Measuring performance

Firefox OS comes with some convenient and easy-to-use tools built-in that, when used properly, can be used to quickly measure performance.

Framerate monitor

The first tool is the "framerate monitor". This can be enabled in the Firefox OS Settings application (Device Information > More Information > Developer > Show frames per second).

The framerate monitor continuously reports three numbers in the top left of the Firefox OS display; the values reported are an average of recent results within a sliding window, meant to be "instantaneous" but fairly accurate. As such, all numbers are "guesses":

• The left number is the composition rate: the estimated number of times per second Firefox OS is drawing frames to the hardware framebuffer. This is an estimate of the user-perceived framerate, and only an estimate. For example, the counter may report 60 compositions per second even if the screen is not changing. In that case the user-perceived framerate would be 0. However, when used with this caveat in mind and corroborated with other measurements, the monitor can be a useful and simple tool.
• The middle number is the layer transaction rate, the estimated number of times per second processes are repainting and notifying the compositor. This number is mostly useful for Gecko platform engineers, but it should be less than or equal to the composition rate number on the left.
• The right hand number is a measure of the number of pixels drawn as a percentage of the screen size. A number of 273 means the screen was painted 2.73 times. Ideally this number should be as close to 100 as possible.

First paint time (time to load)

Firefox OS also has a tool that can help measure startup time, specifically the "first paint" time described above. This "time to load" tool can be enabled using the Settings application (Device Information > More Information > Developer > Show time to load). The value shown by the tool — in the top right of the Firefox OS display — is the elapsed time between when the most recent application was launched, and an estimate of the first time that application painted its UI, in milliseconds. This number only approximates the real "first paint" time, and in particular underestimates it. However, lowering this number almost always correlates to improvements in real startup time, so it can be useful to quickly measure optimization ideas.

For accurately measuring both startup times and responsiveness, a high-speed camera is indispensable. "High-speed" means that the camera can record 120 frames per second or above. The higher the capture rate, the more accurate the measurements that can be made. This may sound like exotic technology, but consumer models can be purchased for a few hundred US dollars. The measuring process with such cameras is simple: record the action to be studied, and then play back the capture and count the number of frames that elapse between the input (say, a tap gesture) and the desired output (pixels changing in some way). Divide the number of counted frames by the capture rate, and the resulting number is the measured duration.

Mozilla built an automated tool called Eideticker, which operates on the same principle as described above. The difference is that Eideticker uses synthetic user input events and HDMI capture to measure durations. The code is available and can be used with any device with an HDMI output.

Measuring power can be more difficult. It's possible to jury-rig measurement apparatus with a soldering iron, but a good approximation of power usage can be gathered by observing CPU load. Simple command-line tools like |top| allow us to monitor CPU usage continuously.

In general, while measuring performance, don't be proud! "Primitive technology" like a stopwatch or logging, when used effectively, can provide eminently usable data.

Diagnosing performance problems

If performance measurements show an application is below its targets, how can the underlying problem be diagnosed?

The first step of any performance work is to create a reproducible workload and reproducible measurement steps. Then gather baseline measurements, before any code changes are made. It seems obvious, but this is required to determine whether code changes actually improve performance! The measurement process selected isn't too important; what's important is that the process be

1. Reproducible
2. Realistic, in that it measures what users will perceive as closely as possible
3. As precise as possible
4. As accurate as possible.

Even stopwatch timings can meet these demands.

Firefox OS includes two built-in tools for quickly diagnosing some performance issues.

Paint Flashing

The first is a render mode called "paint flashing" (Device Information > More Information > Developer > Flash repainted area). In this mode, every time a region of the screen is painted by Gecko, Gecko blits a random translucent color over the painted region. Ideally, only parts of the screen that visually change between frames will "flash" with a new color. But sometimes more area than is needed is repainted, causing large areas to "flash". This symptom may indicate that application code is forcing too much of its scene to update. It may also indicate bugs in Gecko itself.

Animation logging

The second tool is called "animation logging", and can also be enabled in Settings (Device Information > More Information > Developer > Log slow animations). This tool tries to help developers understand why animations are not offloaded to the compositor to be run efficiently as possible. It reports "bugs" like trying to animate elements that are too large, or trying to animate CSS properties that can't be offloaded. The messages you'll get on the device will look like the following:

I/Gecko   ( 5644): Performance warning: Async animation disabled because frame size (1280, 410) is bigger than the viewport (360, 518) [div with id 'views']

Gecko profiler, and other approaches

These tools can help quickly assess a performance problem, but they often show only what developers should be "looking for": hints at deeper problems. When more information is required, the Gecko Profiler can be used. Briefly, the profiler shows mixed JavaScript/C++ "samples" of what all Firefox OS processes are doing over a rolling time interval. Developers can use these profiles to see what their code and Gecko itself are doing. Warnings are built into the profiler UI for many common pitfalls like excessive garbage collection (caused by creating too many JavaScript objects), and forcing synchronous reflows. Sync reflows are bad because Gecko is optimized to do expensive work like reflows in big batches, when needed. Forced reflows throw off this "schedule", and can cause more work to be done than necessary.

Similarly, advanced users may wish to use a whole-system profiler like the Linux |perf| tool. This is mostly useful for platform engineers, though.

As with measuring performance, don't be proud about tools used to diagnose it! A few well-placed Date.now() calls with logging can often provide a quick and accurate answer.

Finally, the only way to keep improving performance is to not regress it. The only way to not regress performance is to test it, preferably with automated tests. A full discussion of that topic is beyond the scope of this document, though.

See also

The Mozilla Paris Firefox OS Performance and Optimization Workshop (March 4-8 2012) included more details discussions of many of the techniques mentioned above. Please take a look at the resources below, which come from this workshop:

1. Technical basics and more
   • Video : Technical considerations
   • Slides : Presentation
2. Performances in a UX point of view
3. Performances measurement & automation